{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initializers\n",
    "\n",
    "Assuming fairly reasonable data normalization, the expectation of the weights should be zero or close to it. It might be reasonable, then, to set all of the initial weights to zero because a positive initial weight will have further to go if it should actually be a negative weight and visa versa. This, however, does not work. If all of the weights are the same, they will all have the same error and the model will not learn anything - there is no source of asymmetry between the neurons. Instead, we keep the weights very close to zero but make them different by initializing them to small, non-zero numbers - this is weight initialization and in caffe2 fillers play the role of initializers. (Ref - datascience.stackexchange) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import all libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.\n",
      "WARNING:root:Debug message: No module named caffe2_pybind11_state_gpu\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Import Complete\n"
     ]
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt            #Plotting the variables\n",
    "import numpy as np                              #Numeric array operations\n",
    "import time                                     #Clocking the cuntions \n",
    "\n",
    "from caffe2.python import core, workspace       #Core creates layers, workspace holds data  \n",
    "from caffe2.proto import caffe2_pb2             #Save layers as files\n",
    "print \"Import Complete\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Filler: Constant\n",
    "\n",
    "Filler details: Generates a blob filled with a constant value. Usually used for filling biases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workspace.ResetWorkspace()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current blobs in the workspace: [u'Y']\n",
      "\n",
      "output_data1 blob shape =  (1, 3, 100, 100)\n",
      "Y:\n",
      "[[[[1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   ...\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]]\n",
      "\n",
      "  [[1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   ...\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]]\n",
      "\n",
      "  [[1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   ...\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]\n",
      "   [1. 1. 1. ... 1. 1. 1.]]]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "op = core.CreateOperator(\n",
    "    \"ConstantFill\",                      # The type of operator that we want to run\n",
    "    [],                                  # Inputs\n",
    "    [\"Y\"],                               # Output\n",
    "    value=1.0,\n",
    "    shape=[1,3,100,100],\n",
    ")\n",
    "\n",
    "workspace.RunOperatorOnce(op)\n",
    "\n",
    "print(\"Current blobs in the workspace: {}\\n\".format(workspace.Blobs()))\n",
    "output_data1 = workspace.FetchBlob(\"Y\")             #Fetching the data back\n",
    "print \"output_data1 blob shape = \", output_data1.shape\n",
    "print(\"Y:\\n{}\\n\".format(workspace.FetchBlob(\"Y\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Filler: Diagonal\n",
    "\n",
    "Filler details: Generates a square blob filled with a constant value in its diagonal. Usually used for filling biases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workspace.ResetWorkspace()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current blobs in the workspace: [u'Y']\n",
      "\n",
      "output_data1 blob shape =  (100, 100)\n",
      "Y:\n",
      "[[-1.5  0.   0.  ...  0.   0.   0. ]\n",
      " [ 0.  -1.5  0.  ...  0.   0.   0. ]\n",
      " [ 0.   0.  -1.5 ...  0.   0.   0. ]\n",
      " ...\n",
      " [ 0.   0.   0.  ... -1.5  0.   0. ]\n",
      " [ 0.   0.   0.  ...  0.  -1.5  0. ]\n",
      " [ 0.   0.   0.  ...  0.   0.  -1.5]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "op = core.CreateOperator(\n",
    "    \"DiagonalFill\",                      # The type of operator that we want to run\n",
    "    [],                                  # Inputs\n",
    "    [\"Y\"],                               # Output\n",
    "    value=-1.5,\n",
    "    shape=[100,100],\n",
    ")\n",
    "\n",
    "workspace.RunOperatorOnce(op)\n",
    "\n",
    "print(\"Current blobs in the workspace: {}\\n\".format(workspace.Blobs()))\n",
    "output_data1 = workspace.FetchBlob(\"Y\")             #Fetching the data back\n",
    "print \"output_data1 blob shape = \", output_data1.shape\n",
    "print(\"Y:\\n{}\\n\".format(workspace.FetchBlob(\"Y\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Filler: Gaussian\n",
    "\n",
    "Filler details: Generates a blob filled with values generated by a guassian function when specified its mean and standard deviation. Usually used for filling weights.\n",
    "\n",
    "Filler Function: \n",
    "\n",
    "$$P(x) = \\frac{1}{{\\sigma \\sqrt {2\\pi } }}e^{{{ - \\left( {x - \\mu } \\right)^2 } / {2\\sigma ^2 }}} $$\n",
    "\n",
    "Standard deviation -> $$\\sigma$$ \n",
    "Mean -> $$\\mu$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workspace.ResetWorkspace()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current blobs in the workspace: [u'Y']\n",
      "\n",
      "output_data1 blob shape =  (1, 3, 100, 100)\n",
      "Y:\n",
      "[[[[-2.7208742e-02  1.8605140e+00  1.3401612e+00 ... -1.9459589e+00\n",
      "    -1.8355805e-01  1.1476872e+00]\n",
      "   [-1.0317578e+00 -9.2503387e-01 -1.6910071e+00 ...  3.2578641e-01\n",
      "    -1.3654517e+00  1.3249282e+00]\n",
      "   [ 3.3635072e-02 -5.2488256e-01  2.3202509e-01 ... -1.8634307e-01\n",
      "    -1.1979079e+00 -1.0280048e+00]\n",
      "   ...\n",
      "   [ 8.8207960e-02  7.0647252e-01 -3.4699577e-01 ... -1.3074943e-01\n",
      "    -1.4389457e+00  6.4721870e-01]\n",
      "   [ 8.1686094e-02  6.0301113e-01  1.4900988e+00 ... -2.0364556e-01\n",
      "     2.0474133e+00  7.7330679e-02]\n",
      "   [ 4.8054972e-01  3.4839222e-03  8.9275575e-01 ... -5.1768380e-01\n",
      "    -2.1811980e-03  1.3668978e+00]]\n",
      "\n",
      "  [[ 9.7451150e-01 -5.9634888e-01  9.0201288e-01 ...  9.3265849e-01\n",
      "     1.3032607e+00  4.5035738e-01]\n",
      "   [ 1.6507663e-01  5.1381654e-01 -1.1267667e+00 ...  2.8291878e-01\n",
      "     6.5497088e-01 -1.4248044e+00]\n",
      "   [ 1.2644882e+00 -1.5056279e+00  1.5272266e+00 ...  3.1550825e-01\n",
      "    -8.0058843e-02  1.7167157e-01]\n",
      "   ...\n",
      "   [-2.8999916e-01  1.1451903e-01  4.2892031e-02 ... -3.6466745e-01\n",
      "    -3.6197248e-01  4.2604268e-01]\n",
      "   [-1.4765109e-01  5.5755109e-01 -4.3774517e-03 ...  1.2451736e+00\n",
      "     6.1665148e-01  2.1281472e-01]\n",
      "   [ 1.6105829e+00 -1.1059363e+00 -1.9377102e-01 ...  7.0613515e-01\n",
      "     6.4738327e-01 -7.9295510e-01]]\n",
      "\n",
      "  [[ 3.8554838e-01  6.9727635e-01  2.2621980e+00 ... -9.7045016e-01\n",
      "     5.1554197e-01 -2.3615928e+00]\n",
      "   [ 1.4436101e+00  4.1594148e-01 -1.6310134e+00 ... -8.2417071e-01\n",
      "     2.1964912e+00 -1.4386165e+00]\n",
      "   [ 2.6058299e-02 -6.3266659e-01  1.1801107e-02 ... -9.0964317e-01\n",
      "    -7.5439829e-01  1.3403673e-02]\n",
      "   ...\n",
      "   [-1.0793796e+00 -4.0298498e-01  4.6788058e-01 ...  4.8233762e-01\n",
      "    -1.1308618e+00  1.4676535e+00]\n",
      "   [ 1.3476539e-01 -5.5186892e-01  1.0785526e+00 ...  6.3451594e-01\n",
      "     8.3761013e-01 -9.2774040e-01]\n",
      "   [-1.5518068e+00  4.0954259e-01  4.4642115e-01 ... -5.8935845e-01\n",
      "     1.3643577e+00 -2.2263863e+00]]]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "op = core.CreateOperator(\n",
    "    \"GaussianFill\",                      # The type of operator that we want to run\n",
    "    [],                                  # Inputs\n",
    "    [\"Y\"],                               # Output\n",
    "    mean=0.0,\n",
    "    std=1.0,\n",
    "    shape=[1,3,100,100],\n",
    ")\n",
    "\n",
    "workspace.RunOperatorOnce(op)\n",
    "\n",
    "print(\"Current blobs in the workspace: {}\\n\".format(workspace.Blobs()))\n",
    "output_data1 = workspace.FetchBlob(\"Y\")             #Fetching the data back\n",
    "print \"output_data1 blob shape = \", output_data1.shape\n",
    "print(\"Y:\\n{}\\n\".format(workspace.FetchBlob(\"Y\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Filler: Uniform\n",
    "\n",
    "Filler details: Generates a blob filled with values generated by a uniform function when specified its minimum and maximum range values. Usually used for filling weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workspace.ResetWorkspace()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current blobs in the workspace: [u'Y']\n",
      "\n",
      "output_data1 blob shape =  (1, 3, 100, 100)\n",
      "Y:\n",
      "[[[[0.3392981  0.4920335  0.12651451 ... 0.26706678 0.23809844\n",
      "    0.7786376 ]\n",
      "   [0.9582259  0.452257   0.53275883 ... 0.9848189  0.59309095\n",
      "    0.2882785 ]\n",
      "   [0.0423877  0.919646   0.44542962 ... 0.2791455  0.2839331\n",
      "    0.60029805]\n",
      "   ...\n",
      "   [0.31680357 0.93511266 0.8756118  ... 0.59136283 0.25160292\n",
      "    0.26761258]\n",
      "   [0.9546697  0.8118115  0.9461675  ... 0.10610344 0.89939827\n",
      "    0.38373634]\n",
      "   [0.8475176  0.8458115  0.16170472 ... 0.51870036 0.11079739\n",
      "    0.07355351]]\n",
      "\n",
      "  [[0.87120175 0.58114386 0.5717167  ... 0.18553035 0.6062569\n",
      "    0.26734623]\n",
      "   [0.6092372  0.6974833  0.9690413  ... 0.9885078  0.67949283\n",
      "    0.48164305]\n",
      "   [0.6854583  0.23517719 0.31614122 ... 0.97349614 0.95112675\n",
      "    0.7181974 ]\n",
      "   ...\n",
      "   [0.15323745 0.11114889 0.45790488 ... 0.40893725 0.19110121\n",
      "    0.24365632]\n",
      "   [0.5102444  0.20981261 0.60793793 ... 0.2815019  0.42414168\n",
      "    0.37863564]\n",
      "   [0.87918913 0.8831598  0.5138676  ... 0.01872852 0.67938805\n",
      "    0.03222848]]\n",
      "\n",
      "  [[0.69934577 0.22825254 0.38999087 ... 0.9913785  0.54365426\n",
      "    0.18111461]\n",
      "   [0.5585565  0.99317694 0.03196645 ... 0.16590145 0.29820427\n",
      "    0.16992374]\n",
      "   [0.43259338 0.5949545  0.4614415  ... 0.14676209 0.80384266\n",
      "    0.40199292]\n",
      "   ...\n",
      "   [0.12445567 0.48591107 0.45858026 ... 0.62989044 0.5905612\n",
      "    0.01947493]\n",
      "   [0.58204824 0.9739704  0.98871386 ... 0.2918398  0.27094585\n",
      "    0.9742284 ]\n",
      "   [0.07405116 0.57341963 0.9808631  ... 0.7603709  0.6281704\n",
      "    0.18973325]]]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "op = core.CreateOperator(\n",
    "    \"UniformFill\",                      # The type of operator that we want to run\n",
    "    [],                                  # Inputs\n",
    "    [\"Y\"],                               # Output\n",
    "    mean=0.0,\n",
    "    std=1.0,\n",
    "    shape=[1,3,100,100],\n",
    ")\n",
    "\n",
    "workspace.RunOperatorOnce(op)\n",
    "\n",
    "print(\"Current blobs in the workspace: {}\\n\".format(workspace.Blobs()))\n",
    "output_data1 = workspace.FetchBlob(\"Y\")             #Fetching the data back\n",
    "print \"output_data1 blob shape = \", output_data1.shape\n",
    "print(\"Y:\\n{}\\n\".format(workspace.FetchBlob(\"Y\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Filler: Xavier\n",
    "\n",
    "Filler details: One good way is to assign the weights from a Gaussian distribution. Obviously this distribution would have zero mean and some finite variance. With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a high value or vanishing to zero. In other words, we need to initialize the weights in such a way that the variance remains the same for x and y ($$y_i = w_i*x_i+b_i$$). This initialization process is known as Xavier initialization. (Ref - https://prateekvjoshi.com/2016/03/29/understanding-xavier-initialization-in-deep-neural-networks/) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workspace.ResetWorkspace()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current blobs in the workspace: [u'Y']\n",
      "\n",
      "output_data1 blob shape =  (1, 3, 100, 100)\n",
      "Y:\n",
      "[[[[ 3.0201944e-03  7.4131228e-03 -5.6186179e-03 ... -8.3116293e-03\n",
      "     7.8045428e-03  2.5014654e-03]\n",
      "   [ 8.1959385e-03 -4.0988540e-03 -1.6142055e-04 ... -7.3929396e-03\n",
      "     5.2170977e-03  9.8491982e-03]\n",
      "   [ 9.8190047e-03  6.9875550e-03  3.9832573e-04 ... -5.0197700e-03\n",
      "    -7.1857339e-03 -7.5253607e-03]\n",
      "   ...\n",
      "   [ 1.8086806e-03 -1.1369744e-03  5.9120338e-03 ...  8.4073562e-03\n",
      "    -9.9160839e-03 -6.4746789e-03]\n",
      "   [-3.5037468e-03  7.0016365e-03 -3.3855508e-03 ... -1.5093265e-03\n",
      "     1.6281828e-03  3.2045748e-03]\n",
      "   [-3.4459378e-03 -5.8587031e-03  2.5901012e-05 ... -2.3018951e-03\n",
      "     6.5810326e-03  8.8225622e-03]]\n",
      "\n",
      "  [[-4.9506794e-03  3.9293896e-03  3.5598027e-03 ... -7.6840324e-03\n",
      "     6.0295481e-03 -1.9379416e-03]\n",
      "   [ 1.3587940e-03  7.7496525e-03  9.3707461e-03 ... -2.6909476e-03\n",
      "     3.2597240e-03  3.1501595e-03]\n",
      "   [-7.7265068e-03 -7.1961493e-03  8.8911932e-03 ... -9.3680806e-03\n",
      "     3.8622143e-03 -3.4846584e-03]\n",
      "   ...\n",
      "   [-6.1211782e-03 -5.7896543e-03  7.5046569e-03 ... -6.9517419e-03\n",
      "    -7.0610736e-04 -2.1802969e-03]\n",
      "   [ 4.8616109e-03 -4.6279989e-03  6.4920355e-03 ...  4.3891845e-03\n",
      "    -5.9151100e-03 -2.3029577e-03]\n",
      "   [ 1.5442660e-03  9.4822869e-03  5.7098418e-03 ... -9.0468880e-03\n",
      "    -8.4458897e-03  4.7589317e-03]]\n",
      "\n",
      "  [[ 4.2306166e-04 -7.0759170e-03  6.9209468e-03 ...  2.3635319e-03\n",
      "    -8.3014872e-03  4.9523367e-03]\n",
      "   [ 2.7364083e-03 -1.8320093e-03 -1.0893131e-03 ...  9.7377971e-04\n",
      "     2.3244144e-03 -4.1116890e-03]\n",
      "   [ 4.4959318e-03 -3.4551704e-03  3.6404375e-04 ...  4.5537325e-03\n",
      "    -3.5635750e-03  7.2868504e-03]\n",
      "   ...\n",
      "   [ 2.3661396e-03  4.5557786e-03  4.4833636e-03 ...  8.2709007e-03\n",
      "    -8.9421449e-03  1.6577682e-03]\n",
      "   [ 2.1822359e-03 -9.2939101e-03  6.8921223e-04 ... -5.2085677e-03\n",
      "     5.3352835e-03  2.3421887e-03]\n",
      "   [-2.2880095e-03  2.8175628e-03  9.7101554e-03 ... -3.7264191e-03\n",
      "     7.8347474e-03 -2.5754045e-03]]]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "op = core.CreateOperator(\n",
    "    \"XavierFill\",                      # The type of operator that we want to run\n",
    "    [],                                  # Inputs\n",
    "    [\"Y\"],                               # Output\n",
    "    shape=[1,3,100,100],\n",
    ")\n",
    "\n",
    "workspace.RunOperatorOnce(op)\n",
    "\n",
    "print(\"Current blobs in the workspace: {}\\n\".format(workspace.Blobs()))\n",
    "output_data1 = workspace.FetchBlob(\"Y\")             #Fetching the data back\n",
    "print \"output_data1 blob shape = \", output_data1.shape\n",
    "print(\"Y:\\n{}\\n\".format(workspace.FetchBlob(\"Y\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Filler: MSRA\n",
    "\n",
    "Filler details: Variant of Xavier filler. Introduced in caffe v1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workspace.ResetWorkspace()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current blobs in the workspace: [u'Y']\n",
      "\n",
      "output_data1 blob shape =  (1, 3, 100, 100)\n",
      "Y:\n",
      "[[[[-0.00677311 -0.01738431  0.01772585 ... -0.0013369   0.00993694\n",
      "    -0.03332539]\n",
      "   [ 0.02923565 -0.00170157 -0.01091725 ...  0.00872084  0.01478975\n",
      "     0.01710992]\n",
      "   [ 0.00827969 -0.00756499  0.01339058 ...  0.01961535 -0.00961402\n",
      "     0.01514838]\n",
      "   ...\n",
      "   [ 0.00607628  0.0011068  -0.01788989 ... -0.0165804  -0.00708509\n",
      "     0.01785469]\n",
      "   [-0.01042639  0.01188143  0.007033   ... -0.00706194 -0.00490387\n",
      "    -0.00523129]\n",
      "   [ 0.00816684  0.0034749   0.00101617 ...  0.0122927   0.00624652\n",
      "    -0.01117847]]\n",
      "\n",
      "  [[-0.01427509 -0.00733254  0.00021133 ...  0.00169657  0.00642907\n",
      "    -0.00139695]\n",
      "   [-0.01733812 -0.03446689  0.00583131 ... -0.00047469  0.002245\n",
      "     0.00871252]\n",
      "   [ 0.02446129 -0.00768537  0.00846224 ...  0.00041484  0.0041176\n",
      "     0.0160145 ]\n",
      "   ...\n",
      "   [ 0.0184339   0.0031943   0.00386364 ... -0.00246572  0.01736321\n",
      "    -0.01589323]\n",
      "   [ 0.02079595 -0.0103161  -0.00292767 ... -0.00947183 -0.00096688\n",
      "    -0.01967942]\n",
      "   [-0.00420307 -0.01329974  0.00320903 ...  0.00114059 -0.01362096\n",
      "    -0.001236  ]]\n",
      "\n",
      "  [[-0.00424407  0.00163034  0.02584586 ... -0.00917584 -0.00667131\n",
      "     0.00810445]\n",
      "   [ 0.00101916 -0.00327092 -0.0100486  ...  0.00947021 -0.01564067\n",
      "    -0.00177132]\n",
      "   [-0.00091399  0.01153107 -0.00779413 ... -0.00040511 -0.0165381\n",
      "    -0.00144171]\n",
      "   ...\n",
      "   [ 0.00437844  0.01619907  0.02688778 ... -0.02048137 -0.00086056\n",
      "    -0.00970437]\n",
      "   [ 0.00026684 -0.01439685 -0.0136652  ... -0.00974845 -0.0023584\n",
      "     0.01039742]\n",
      "   [-0.00145364 -0.00198072 -0.00130647 ...  0.00811371  0.01439357\n",
      "    -0.01498089]]]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "op = core.CreateOperator(\n",
    "    \"MSRAFill\",                      # The type of operator that we want to run\n",
    "    [],                                  # Inputs\n",
    "    [\"Y\"],                               # Output\n",
    "    shape=[1,3,100,100],\n",
    ")\n",
    "\n",
    "workspace.RunOperatorOnce(op)\n",
    "\n",
    "print(\"Current blobs in the workspace: {}\\n\".format(workspace.Blobs()))\n",
    "output_data1 = workspace.FetchBlob(\"Y\")             #Fetching the data back\n",
    "print \"output_data1 blob shape = \", output_data1.shape\n",
    "print(\"Y:\\n{}\\n\".format(workspace.FetchBlob(\"Y\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Effect on training with different initializers (fillers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.\n",
      "WARNING:root:Debug message: No module named caffe2_pybind11_state_gpu\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Necessities imported!\n"
     ]
    }
   ],
   "source": [
    "from matplotlib import pyplot\n",
    "import numpy as np\n",
    "import os\n",
    "import shutil\n",
    "import caffe2.python.predictor.predictor_exporter as pe\n",
    "from caffe2.python import (\n",
    "    brew,\n",
    "    core,\n",
    "    model_helper,\n",
    "    net_drawer,\n",
    "    optimizer,\n",
    "    visualize,\n",
    "    workspace,\n",
    ")\n",
    "from IPython.display import Markdown\n",
    "from ipy_table import *\n",
    "# If you would like to see some really detailed initializations,\n",
    "# you can change --caffe2_log_level=0 to --caffe2_log_level=-1\n",
    "core.GlobalInit(['caffe2', '--caffe2_log_level=0'])\n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "print(\"Necessities imported!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**lmdb train db found!**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**lmdb test db found!**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "Looks like you ran this before, so we need to cleanup those old files..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**data folder:**/home/abhi/caffe2_notebooks/tutorial_data/mnist"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**workspace root folder:**/home/abhi/caffe2_notebooks/tutorial_files/tutorial_mnist"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Download Data\n",
    "\n",
    "# Adopted from: https://caffe2.ai/docs/tutorial-MNIST.html\n",
    "\n",
    "def DownloadResource(url, path):\n",
    "    '''Downloads resources from s3 by url and unzips them to the provided path'''\n",
    "    import requests, zipfile, StringIO\n",
    "    print(\"Downloading... {} to {}\".format(url, path))\n",
    "    r = requests.get(url, stream=True)\n",
    "    z = zipfile.ZipFile(StringIO.StringIO(r.content))\n",
    "    z.extractall(path)\n",
    "    print(\"Completed download and extraction.\")\n",
    "    \n",
    "current_folder = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks')\n",
    "data_folder = os.path.join(current_folder, 'tutorial_data', 'mnist')\n",
    "root_folder = os.path.join(current_folder, 'tutorial_files', 'tutorial_mnist')\n",
    "db_missing = False\n",
    "\n",
    "if not os.path.exists(data_folder):\n",
    "    os.makedirs(data_folder)   \n",
    "    print(\"Your data folder was not found!! This was generated: {}\".format(data_folder))\n",
    "\n",
    "# Look for existing database: lmdb\n",
    "if os.path.exists(os.path.join(data_folder,\"mnist-train-nchw-lmdb\")):\n",
    "    printmd(\"**lmdb train db found!**\")\n",
    "else:\n",
    "    db_missing = True\n",
    "\n",
    "if os.path.exists(os.path.join(data_folder,\"mnist-test-nchw-lmdb\")):\n",
    "    printmd(\"**lmdb test db found!**\")\n",
    "else:\n",
    "    db_missing = True\n",
    "\n",
    "# attempt the download of the db if either was missing\n",
    "if db_missing:\n",
    "    print(\"one or both of the MNIST lmbd dbs not found!!\")\n",
    "    db_url = \"http://download.caffe2.ai/databases/mnist-lmdb.zip\"\n",
    "    try:\n",
    "        DownloadResource(db_url, data_folder)\n",
    "    except Exception as ex:\n",
    "        print(\"Failed to download dataset. Please download it manually from {}\".format(db_url))\n",
    "        print(\"Unzip it and place the two database folders here: {}\".format(data_folder))\n",
    "        raise ex\n",
    "\n",
    "if os.path.exists(root_folder):\n",
    "    printmd(\"Looks like you ran this before, so we need to cleanup those old files...\")\n",
    "    shutil.rmtree(root_folder)\n",
    "\n",
    "os.makedirs(root_folder)\n",
    "workspace.ResetWorkspace(root_folder)\n",
    "\n",
    "printmd(\"**data folder:**\" + data_folder)\n",
    "printmd(\"**workspace root folder:**\" + root_folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constant hyperparameters\n",
    "\n",
    "arg_scope = {\"order\": \"NCHW\"}\n",
    "\n",
    "def AddInput(model, batch_size, db, db_type):\n",
    "    # load the data\n",
    "    data_uint8, label = brew.db_input(\n",
    "        model,\n",
    "        blobs_out=[\"data_uint8\", \"label\"],\n",
    "        batch_size=batch_size,\n",
    "        db=db,\n",
    "        db_type=db_type,\n",
    "    )\n",
    "    # cast the data to float\n",
    "    data = model.Cast(data_uint8, \"data\", to=core.DataType.FLOAT)\n",
    "    # scale data from [0,255] down to [0,1]\n",
    "    data = model.Scale(data, data, scale=float(1./256))\n",
    "    # don't need the gradient for the backward pass\n",
    "    data = model.StopGradient(data, data)\n",
    "    return data, label\n",
    "\n",
    "def AddAccuracy(model, softmax, label):\n",
    "    \"\"\"Adds an accuracy op to the model\"\"\"\n",
    "    accuracy = model.Accuracy([softmax, label], \"accuracy\")\n",
    "    return accuracy\n",
    "\n",
    "def AddModel(model, data):\n",
    "    return AddLeNetModel(model, data)\n",
    "\n",
    "base_lr = 0.001\n",
    "m_high = 0.9\n",
    "def AddTrainingOperators(model, softmax, label):\n",
    "    xent = model.LabelCrossEntropy([softmax, label], 'xent')\n",
    "    loss = model.AveragedLoss(xent, \"loss\")\n",
    "    AddAccuracy(model, softmax, label)\n",
    "    model.AddGradientOperators([loss])\n",
    "    optimizer.build_sgd(\n",
    "        model,\n",
    "        nesterov=1,\n",
    "        momentum=m_high,\n",
    "        base_learning_rate=base_lr,\n",
    "        policy=\"step\",\n",
    "        stepsize=1,\n",
    "        gamma=0.999,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Constant initialization - Very High value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Training status: Running**  Wait till the process is completed"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Completed iterations: 0 , Total iterations to be completed: 2000 , Current Loss =  43.17347\n",
      "    Completed iterations: 100 , Total iterations to be completed: 2000 , Current Loss =  43.17347\n",
      "    Completed iterations: 200 , Total iterations to be completed: 2000 , Current Loss =  37.417007\n",
      "    Completed iterations: 300 , Total iterations to be completed: 2000 , Current Loss =  41.734352\n",
      "    Completed iterations: 400 , Total iterations to be completed: 2000 , Current Loss =  41.014793\n",
      "    Completed iterations: 500 , Total iterations to be completed: 2000 , Current Loss =  41.734352\n",
      "    Completed iterations: 600 , Total iterations to be completed: 2000 , Current Loss =  39.57568\n",
      "    Completed iterations: 700 , Total iterations to be completed: 2000 , Current Loss =  43.17347\n",
      "    Completed iterations: 800 , Total iterations to be completed: 2000 , Current Loss =  41.014797\n",
      "    Completed iterations: 900 , Total iterations to be completed: 2000 , Current Loss =  35.977894\n",
      "    Completed iterations: 1000 , Total iterations to be completed: 2000 , Current Loss =  42.45391\n",
      "    Completed iterations: 1100 , Total iterations to be completed: 2000 , Current Loss =  41.734352\n",
      "    Completed iterations: 1200 , Total iterations to be completed: 2000 , Current Loss =  41.734352\n",
      "    Completed iterations: 1300 , Total iterations to be completed: 2000 , Current Loss =  43.17347\n",
      "    Completed iterations: 1400 , Total iterations to be completed: 2000 , Current Loss =  38.136566\n",
      "    Completed iterations: 1500 , Total iterations to be completed: 2000 , Current Loss =  41.014793\n",
      "    Completed iterations: 1600 , Total iterations to be completed: 2000 , Current Loss =  42.45391\n",
      "    Completed iterations: 1700 , Total iterations to be completed: 2000 , Current Loss =  42.45391\n",
      "    Completed iterations: 1800 , Total iterations to be completed: 2000 , Current Loss =  42.45391\n",
      "    Completed iterations: 1900 , Total iterations to be completed: 2000 , Current Loss =  39.57568\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Training status: Completed**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Testing status: Running**  Wait till the process is completed"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Completed test set: 0 , Total sets to be tested: 1000\n",
      "    Completed test set: 100 , Total sets to be tested: 1000\n",
      "    Completed test set: 200 , Total sets to be tested: 1000\n",
      "    Completed test set: 300 , Total sets to be tested: 1000\n",
      "    Completed test set: 400 , Total sets to be tested: 1000\n",
      "    Completed test set: 500 , Total sets to be tested: 1000\n",
      "    Completed test set: 600 , Total sets to be tested: 1000\n",
      "    Completed test set: 700 , Total sets to be tested: 1000\n",
      "    Completed test set: 800 , Total sets to be tested: 1000\n",
      "    Completed test set: 900 , Total sets to be tested: 1000\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Testing status: Completed**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7ee1cd17f3d0>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8FdXdP/DPIQGCgCwB2VcBq2UJIQoURQUVxQVRq4gV3J7WtjzVlsentNpW7a+17vqodUVxBcUNEWUpihYsStiXiCyCBIGEQNhDQvL9/fG9w8y9mbl37s3NTTJ+3q/Xfc3cubOcOXPmO2fOzJ0xIgIiIqr76tV0AoiIKDkY0ImIAoIBnYgoIBjQiYgCggGdiCggGNCJiAKCAZ2IKCAY0ImIAoIBnYgoINJTubBWrVpJ165dU7lIIqI6b+nSpbtFpHWs8VIa0Lt27Yrc3NxULpKIqM4zxmz1Mx6bXIiIAoIBnYgoIBjQiYgCIqVt6EQUXGVlZcjPz0dJSUlNJ6XOysjIQMeOHVG/fv2EpmdAJ6KkyM/PR9OmTdG1a1cYY2o6OXWOiKCoqAj5+fno1q1bQvNgkwsRJUVJSQkyMzMZzBNkjEFmZmaVznAY0IkoaRjMq6aq+feDCOgLFgDr1/sff/9+4I03/I9/6BDw2muxx/vmG+CTT8KHFRUBb7/tPc2sWUB+fvT5fvAB8P330cf57DPg669jp7G2O3oUmDIFcL458Z13gMJC72kqKoCXXgLKyir/tnYt8O9/hw9bvBhYuVK366uvxk6TCPDKK8Dhw75WocYtWQIsW1bTqahepaVAcXHNLb+8XPftlBORlH0GDBggNUF3Of/jX321jr98ub/xb7xRx1+4MP50DB2qw77/3nuatm1jz7dLl/iXXRdNmqTr8c47+r2wUL8PHuw9zSuv6Dh//Wvl39zyxRp2ww3aXbQoepr+9S8d75e/jG9dakp1lYV169Ylf6Zxaty4sYiIrFghsmRJzaVj40Zd/qFD8U/rlo8AcsVHjP1B1NDjtX27dg8e9De+VYP2O77Tli3aLS31HmfnTu/frJrqVl//I6v7du3S7t692j16VLvR1n/PHu1Gq8W72bZNu4cORR9v3z7tRttOlFpuZ2OpZO3PFRWpXW6dDejbtwMbN2r/gQP2KeThw3pK6Ud5ObBwYeXh9UK5UlEBHDtmj7Nkic7fazqv5q916/wHE+e6OH3zDbBjR+XhVSkwe/dq00JViQCffx7eDOLX/v3A1KkaqJcs0eC5di2we7f7+FYer1qlXWv9vfL++eftg4Cf9Dnz3pp3rGtU1nyN0e00bVrl3xPNn1277KayhQsT39779gErVtiVFcvSpVrmAC0Lbs0UVvqdvv469gFMROedyHoDerC2DtheysqAI0eij7N27RYMGzYMffv2xfDhw/Hdd98BAKZPn47evXujX79+GDp0aGjctTjjjDOQlZWFvn37YsOGDWHzcq6PFQus/mPH4l7F5PNTjU/WJ5lNLs7TRqvZoqxM5Mortb+w0H1cp3vv1eGffRY+/Oyzdfinn4rcdZf2f/ihdq+4QuS++7R//nwd/7zz9PucOd5pbdbMPR2dO+uwLVvC53XkSHjavdahtNTfKbTbOP37J+fU+6WXdD6vvx7/tFZeW58rrtBuhw7u4998c/i6bN2q/Z06VR539uzwef/3f1cexzkvZ14CIn36aLdFi+jrMH26jmeVPUDTZZk6VYdNnhwzOypp0ECnnTtXuw88EP88RER+8pPKZenwYe2OGKHjAFomIj31lP723nv2MEDEmPDxnE0Ft92my8zOFhkyRLdzvJ/sbP1Y32+7rXLali0Lb1qxmlyWLNHP0aMiZ555iTzwwBQREZk8ebKMGjVKRER69+4t+fn5IiKyd+9eERGZMGGCvPbaayIicvToUTl8+PDxeRcX6zx37BCpqND+r7+W48tbvdpOR16eDjtwoHKaY/nBN7lYtYeKCuCrr7Q/1mkyAKxZo93IC4rOGvrq1dq/ebN2Fy/WGjdgn5Jbol2gtk7LY/niC+36PdpXpYa+fLl2E61BWawzpU2b4p828oLkl19qN7ImaYnM42g1dKs5y6/I03SrCc1q3vHirKFbnBfErLITUdnzxTp1t5qU4rm472SVKyfrIq61zwB2mXCyzhAim7VilRtr21Rns4NVQ472++rV/8F5540FAFx//fVYGDq9HjJkCG644QY8//zzKA/NaPDgwfj73/+O+++/H1u3bkWjRo2Oz8vaFiUl9nKdccbtTK6q+1a86tQfi8rLgYICoF07e5hzxykp0d8BPWU/ehTo1St8HhUVdtNFZGYXFOhObAX01avtU9D0UE59/719imcVVCuwFxXpvE86SbsdO7qvx+HDmtaWLcOH79plF5ojR9wLa34+0KGDBo9vv/Uu0MXFmuYTTqh84AGAvDy7v6wMaNAgfBnOtJeV6finnqrpEgGaNdP137nTDmQiGvwKCoBTTtFtkJ6u07d2PPhz+3agfXvdPpE7u1cgB3TezjtJSkrspixjdNoOHezfI7dvZBOB8/vevZWbG9yuiRQX63Zq3Fjz6OBBu43e2SS2c6fmVaNGdnnKy9O8KCoCWrWyyxQQvl3dWNu5no8q2NGjGoR799Y88mq2sA4w9etXzpt9+3RZBw7YFZhYyxbR9atfH3j0UZ3//v26rVu1Ci9jkY4d0/xq3Fj769fX5iAAyMlxH9+r0uMcbu1PVvNNWZl9jvLPfz6DhQu/xJw5szBgwAAsXboUY8eOxcCBAzFr1iyMHDkSzz77LM46axjS0uxtU1pql9t69cIrAocO6TaPdaCpLnWqhv7732vhcLavtmpl948ZYxfenBwNKpE1wH/8Q3fEjh2B6dN1mLWh2rQBTj4ZmD9fv99+u97uBwBpafY8rNsMKyp0Hlat6dprNX29ewOdOgEzZ7qvR1YWkJmp/VbQEQHatrULY1YW0Lx55Wk7dQJeeEFrfd27Az17ui+jRQugc2fg738HIh9Bv3AhcNpp9nfnBdkvvtBlvPKKPez664F+/YDx4zWQW+m65x4NQtZFYRE9SP3oR9ou3rq1puOkk+x55eVp3j/6KHDRRe5p99KyZfjtpI0aAWecof1bt+p8nWcJkQeL6dPtIH34cHjFoGVLzU8n53UP65pJixa6fp06aTnp3Bm49Vb9bdEie/yRI4FBg7TfKjszZmgZbtcO+PWv7XGXLdP5Pfus97pbAcJZDr2cc46WnzvuAPr00bLmFowHD9ZuZCUJ0G184om6fa39IdayCwvtazKFhRrMAa0ErVoV/axzxQrdj5Yt03G9rqE4x7fOsJ0qKvQ3y4YNQN++P8HcudOwejXw4IOvo2/fs5CfDyxcuAknnDAQo0ffi+bNW2Pbtm3YvHkzunfvjt/85jcYNWoUVq1ahZUrw8+u9u+3z+qNCb8OlZeny4/Vrl9d6lRAf+897XrdX/rxx5WHWUd5y6xZlccxJvZpoVvNqaLC/eKodYrqdXHW7dQ78oge7b7yxYtj35sOaK1zzpzKwyN3BGdAX7tWu86LYG++qd2pU8On++gj7VpnRc48jDzFtw5cVsCdP1//H5Bszlqy2za1TpFjXWyL5BY8Vq2K3hxjXbh1lh2r/FllGbDPliIvPDpZwTDdxzn14sXafecdPYsD/NXsY4k1D2cQs4K5Uzy11kTuGAOAkpLDuPjijsc/r7/+CO644wnMnPkSrr22Lz766FVMnPg4ioqAe+65A2PG9ME11/RGVtZP0K9fP7z11lvo3bs3srKysGbNGowbNw6Anqk4t6PVhOq2Tqm+s8WpTjS5FBdrgbYyL9otfpEiM9xtWmMSu81JpOqnVs7mCr8aN/ZXU4u1TIszuFnz9bNeVsG1pnGuQ2QN5cgRbf5JhNV04UfDhpXT52Ste7wBzi2/ozUhODmX1bixdq2yU1Zmt7060xu5zvHU0C3O7RrP+nqVRb/z8Jo+nkCX6B8mv/rKfSFPP/1JpWGPPPLu8f2+WTNd5qRJkzBp0qSw8axrIE7WOtZk8HZjJIWt9jk5OZLIG4suugiYPbvqy7//fmDyZL21zKlHD/vCXjyefhr45S+9f7/lFm0e8ePhh4GJE/0v+4Yb9B+Tydali33xa8uWys01NWXNGm3K8ksEeOwx4Le/Tc7yX3hBt2cyjRxpn+U4vfsucMUV3tMNGgT85z920Gvb1v0WwhNPdK8pV4VI5Yu/1rWg2bPz0KrVqb7m06iR+zUUS2amfX0sJ0fb8a3rJ6edZrfrVwev/EyU2zWAaPLy8nDqqeH5aIxZKiIx51QnmlwSfJJkJX/9q30ByymRYA7ErlXH81f7yHuXY6mOYA6E38lQnTtNvPw0MUVK5oWpZDRZRHIL5oB7M5mT1aRi8Qo+qbgvOpHtAugZSLTabeRvzovhfu8YS1Rd/oPYDyqgG5PcixWxTrfiacaJpxkpVRJtIqkOzqYUv5J5OlwdAd1LsgJxKgJ6VZr+oonW5FLbmjlqkzrR5DJmjH1hriqaNg3/dxcF1/Dh9t0ZlBoff+y/yeWHhE0uEZJVQ3deWKVgYzCnH6IfVECvjc0aRETJUicCerKeK8zaOVHwLVjwPk4/3WDLltrxAoDcXP2k4lWrdSKgu/15h4jIzZw5U5GVdSbmzJkae+QElSdQO0zFy5zqREBP5A4HIvrhOXz4IFauXIi77pqMuXPte4Fffvl+jBnTB2PH9sMTT+gfh7Zt24hf/eo8jB3bDz/7WTby8zdh6dIF+O1vLzk+3QMPTMDMmVMAAJdd1hVPPPF7/Oxn2Zg/fzree+95jBt3OsaO7Yf//d8rUVKi91YWFe3CHXeMxtix/TB2bD+sXPkFnnnmz3jyyceOz/fOO+/E448/nvT1rxP/FGVAJ6pbOj18O074ZkXsEeNwuFcWtk18LOo4n302A4MHX4guXXqhWbNM5OUtxZ49BfjssxmYMuVLZGScgH379M8of/rTdRg/fhLOPXc0jh4tgUgFdu1yeZKdQ7NmmXjtNX1ofnFxEUaP/i8AwNNP34UZMybjmmv+Gw899Bv07382HnzwPZSXl+PIkYNo3bo97r77CkyceDsqKiowbdo0fOV8zGWS1ImAnpFR0ykgorpg7typGDPmNgDABReMwZw5UyEiuPTSG5GRoX+saNasJQ4dOoDCwu0499zRAICGDf0FmfPPv+Z4/6ZNa/DMM3fhwIFiHDlyEIMGjQAA5OZ+gnvu0afbpaWloUmTZmjSpBkyMzOxfPly7Nq1C/3790em9YS+JPId0I0xaQByAWwXkUuMMd0ATAOQCWApgOtFpFruI/HzQKJkEgHuvRf4y19Su9wgGDVKnyqYKtOnAz/9aeXhjzwC/O53qUtHsrk9Y70uiVWTTrbMTGDz5j1YsuQTbNy4GsYYlJeXwxiD4cNdCgj07rm0tPCbJdLS0lHh+OdSaWn4lcxGjRof77/33hvw4IPvo1evfpg5cwqWLl0QNY233HILpkyZgp07d+Kmm26KfyV9iKcN/TYAjqdo434Aj4pIDwB7AdyczIRR3ZTq5rFUv0CAaidjgPnz38bIkddj5syt+OCDLZg1axvat++GJk2aYebMl463ce/btweNGzdFx44dsWDB+wCA0tKjKCk5jLZtu+Dbb9ehtPQoDhwoxpIl3n9oOHToAFq1aodjx8owe/brx4effvpwvP320wD04unBg/qsgtGjR2P27NlYsmQJRowYUS354CugG2M6ArgYwAuh7wbAMAChJ4PjZQCXV0cCdXnVNWdv8T6r243fp/EFSW253nFqDf1hsU+fxKbz+xA3cpeWps0t55wzOmz4sGFXoqhoB4YOvQzjxuVg7NgsvPbaQwCAV199FdOm/R+uvbYvbr75J9i9eyfatu2E8867GmPG9MYf/nA1evXq77nMW2/9K268cSBuvnkIunb90fHhEyc+jqVLP8WYMX1w/fUDsHmzPhSpQYMGOPfcc3H11VcjrZqemeDrr//GmLcB3AegKYD/AXADgMWh2jmMMZ0AfCwiUZ+Hl+hf//v1s58tffbZ9ksnIv3zn8CvfhX37Cvxc7pbWBj+Fh5Lkyb2s5xXrNC01+bT5ubNvZ8vf+aZ7reMnnsu8Omn7tNU11Mgvbz5JnDNNZWHf/EF8JOfpC4dlsinEfrxwAP6MgrndH6bXK6/Xl9GEjne736nzU7xGjFCn2yaSJmtyl//c3J0v3E+0C4nR+/f9sP5lFCnnj29X/0X+RTHE07QJzl++63//75062Y/cz6W7OwKZGdnY/r06ejp9WYaVPNf/40xlwAoEJGlscb1mP7nxphcY0xuofMVMHHNw+5PdXu6F690ONNa1x8ilMgDqVL5EKtoavNBNFJVKmte+Z3oflJXm7ASLXfO6az+ePLA77ibN69Djx49MHz48KjBvKr8ZMMQAJcZY7ZAL4IOA/A4gObGGKvYdATg+jZIEXlORHJEJKe1W5XWh5sdrfO1JaD72Qmt93IOGBB7XOcr4fzo1Utffxbpuuvim080V17pPjxaIU51QO/vcUbcrVvlvGjRwns+3bolL0233Rbf+JFlqU2bxKe1pCKgX3ZZ5WGNG1ce5ldVDsJeTX1uL0Yxxn6SqPNGE6sC5vbqRy9+17d799OwefNmPPzww/5nnoCYu5+I/EFEOopIVwBjAHwiItcB+BTAVaHRxgOotnsbJkwAXnxR+90KasOGujGcBeKdd7R7zjmJL/fOO7V7zz36cgynWDvMjh12k4yfuz7WrInv0QQrV+rr3A4cCB/+2GOaF0884X9eXi67TNP0zDPhw6uy45WXAwMHuv+Wna1/j/bzF+mdO3VePXtWPhOqqNCg+NBD4cPfeUenqagAQm8WO75c57tII5+H/Ytf2P3WM+KjvUHp0Ud1GdG25y232M2DkUHZ+Ro95/tY3VjTWsv705/0e7Ty6fZYXesZ7M438cQqjzNmhL8VqVMnoFcv+4jgVpHx80jmBg10m3hxO4hbtzY7KxSdOrlfx8rOtq+xOIN3kyaVh0WTnR1eDqzpq6KqT7+tSn3q9wB+Z4zZCL11cXKVUhKFMdFrfsZUDjLW92S8qs1t+bECunN8P7WlWOsYKSND5xtZiOrXd8+PRFhpikxXtHnHWm69et7bJC1ND85+Lqw60+W17SOHi+g0xoQHovT08HEj19eZXmtbRtumVv7HKrNer5VzpiXWM/Wtaa3lWQe3aA+0c8v/yOYGv+XRuZyMjAzs3VsEQI7PIx7R9jcnt/THsyyv/cOr3ESbTzKJCIqKipBRhT/exHViJiILACwI9W8GcEbCS06QdcR1XoywCqHzFWU9emj3ggv0UaonnxxeC3PTsqX7G42AyhvPKyi5vbMyntPfDh2A7a6NV97OPx+YN0/7rWDo526Lvn21lhF5EfOqq4C339ZXmAHxFVw/FYwLLqj8EunI5fTqpa8KHDPG/W1OftIUeTrcqZPdP2SI/Yz9yHlFTjdokF5wB+ya4BVXAC+/HD5ePK/rGzxYy8izz0bfVqNGRb/IHFkOnS+TbtDA/xNGu3TR7rBhscdt1Mh+UYwz7zp06IgdO/JRUFCI+vX15de7d4dP65WmvDw9eO3erWm39pnI6a1xI4evX6/DGje2XwSelqb7c+S4eY6br8vL7d+PHrVvaNi9O3w9Gzas/GJx6wKuNX1ZmZ4xV1RoOYk803Qu10tGRgY6Wm21iRCRlH0GDBggiZoyRQQQGTdOJD9fpKxMZN06HdaggT3eunUiBQXav2WLSEWFyPbtIkePipx1lo7v/CxaZPfv3y9SVGTP6667dPi994o89FD4dCJ2/3ffiezZI1JcLNKsmQ5zzqe4uPJyIz+WgwdFJk+OPu7ateF5c+SIpmH79vDhW7aI7N7tPo8//lHk0CGR0lKRlStFdu4U+eorkW3bNG/z8+35vPBC+LTDhmm3VSuR5ctFune3f7vlFrv/o49E1qwRKSkJX8/yck3v5s0iX3xh/zZwYPg65eaKHDum40amf/fu8HV1y0sRkQ0bRObOFVm2LHx4RYXIiy/q+IMGhc9DRJf57beah87fioo0n0tLw5dZWChy4IBU4hznX//ScmKVy4oKe/6Ry7eUlorMmOFdFn7zm/DxJ07U4Q8+KLJvn6bXray5Ddu6VbeNV/qtz4YN9j7mHKeiQr878yJyupwc7X/6aS1zmzaJ7N2r4+bl6W89e7ovf9s23Ze80v/997qfFxeLrFjhvQ5OzvyZONEevmNHeLk9fFhk1y5Na+R8iotFVq3Ssrp/v5bNI0d0/fbtc19uvADkio8YW0suMcanQwftdu+uXXHUCp13+1i1jvbttdu3L/Dvf4fPy3lrW9OmiaXHWfOzOGsusZp9nDXCxo1jXyCNvBiakeGeBmv93bRqZbdn9u2rXefFOCuPAe/T36ZNgays8N+d28LrXv569ez0el2MzMiw22Bj5W80PXrYZ2uR0/fqZacnktsyrXGt8uTUqlXstDRtqhdmnRdno20jQJs02rXz/j2ybFlNNOnp9hmWX24X2d00bep+y661TbzyokcPu3zk5Hhf/PUqb7EqrlY+NWigtwv74SxHzv62bcPHa9TI+7pJs2b2WZYzhtTEI0tqyU1msVkb85RT7GHWadmgQf7mYe3A1lvK45FoeyAQu8nl7LPDv1s7/ODB2o28nzoZbXfOwBuLtTzrAp0VaK0LV84dsCp3ZMXbDlpVVnuzc15uwTpZy43n7gknt7tzrHRG5vfJJ2s31l07zv0oXvHkwemnh3+3Aq3bOlnbI947pfwcTL14BfRorApQbVRnaujDh+ufWc46yx6WlgYsWeI/iEyYoEfSV18FXnrJHv7NN7GPptbGzs72968+Z8CMVkNfvBj48Y/Dh51yiv6hJycH+OorvStk8WJtKy8tTX1At3awESOAm24Chg7VNmSrVmL9/s9/6h0hkyYllqZk3PIY7e6TSFYeWMtduTJ2QHfm25IllQOWl4UL7QqFl+3b3d9o36MHMGsWcPHF9rArrwQuv1z/5OVklXHn8C1btG29Xj07/YsW6R07336r5crL8uVac1+7Frj0Uvf0RTN3rpZdq0L21FN6G7J14HFyO8Ba1qxxn/+qVbFv89y40f0sLXJZbsvdurVy2/mCBf7/TARofEnVP6jrTEAH3G9BjOcFrPXqaUF/7bXw4X4OCNbGPvNM73ufneM5RQb0a68Fpoaeve91C9+QIdq1DmBDh+rpd2lpfMHYSyI19IoKexs4z4qs34cOrVpQTsaBKp77oCMDSLSaV7NmGsyc+RZPTc3antG0b+99QLHO1pzcLmBaZdzJrVknM1PLlrOC5CYrS7tnnWXfkBBP2WneHLjwQvt7Rob3P3gjD7BOkZUei5+L/24HD0usgO7WDBXZbBZLNf6PqJI60+SSTLHu7bVYfzpo0cJ/sLHudHAG8cgC6nf5kazT6GTUZOM5/bfG9Uq3FTCqWguJVTt28goq8TQlWMHfeb3Ai5X30bZrdYqsFCTjoF7bWOXH6/pFdYh2u2pdVKdq6Mly9916e9GYMdHHmzBBd/qbbgKefFKHxdqRZs8GPv+88gWp99/XC1Zdu2pto0kTf7U2p3nz9FQ5nmYFN889B9x4o//xL7lEm5nGjnX//Y03gI8/tk9rZ8+2L1hbNm70fqZGbq7+cey++/ynycv77/sf9/TT9dbD0aNjjzt7tl5Qd27XVAaAE0/URwWvWAH87W81G9AjKzeff65nMFXVq5eePY8caQ9budL9WUMbNmiZiteyZeHfgxbQY94Gk8xPVW5brGmPP663Hk2YYA9Lxu1IqRDttq26wkp7ixb27YNuv9dUuhL9PV5PPKHz+/WvkzdPv1q31mXv3Jn6ZVeV13Y4eND+7a67Up8uv+DztsUgHJNSwjodrC2Ph/2hSsbfq5Mtlc8Xstqxa+LRzNYteXXpwWexONclCPv2D7LJJRE33qhXtq3nuwD6GF+3R3bWNgsX6ilq06a1MyD6sXgxsHq1XpR9993Ebj2tDo89pndgeXnuOf/3RPtxww3A5s3h5TBV5s7VZp9ErwHVpI8/tv8F6uQM6BMnpi491cXX89CTJdHnoRPFYu2YQbxYSNWnpMS+JlWby07SnodORBRUQWo+AtjkQgHx4ovAtm01nQqqa+J9wmJtx4BOgRDPbZhElqAFdDa5ENEPHgM6EVEdl56uT1Z8/vmaTklysMmFiH6wjAl/5V9dxxo6EVFAMKATEQUEAzoRUUAwoBMRBQQDOhFRQDCgExEFBAM6EVFAMKATEQUEAzoRUUAwoBMRBQQDOhFRQDCgExEFBAM6EVFAMKATEQUEAzoRUUAwoBMRBQQDOhFRQMQM6MaYDGPMV8aYlcaYtcaYe0LDuxljvjTGbDTGvGmMaVD9ySUiIi9+auhHAQwTkX4AsgBcaIwZBOB+AI+KSA8AewHcXH3JJCKiWGIGdFEHQ1/rhz4CYBiAt0PDXwZwebWkkIiIfPHVhm6MSTPGrABQAGAegE0AikXkWGiUfAAdqieJRETkh6+ALiLlIpIFoCOAMwD8yO8CjDE/N8bkGmNyCwsLE0wmERHFEtddLiJSDOBTAIMBNDfGpId+6ghgu8c0z4lIjojktG7dukqJJSIib37ucmltjGke6m8E4HwAedDAflVotPEAZlRXIomIKLb02KOgHYCXjTFp0APAWyLyoTFmHYBpxpj/B2A5gMnVmE4iIoohZkAXkVUA+rsM3wxtTyciolqA/xQlIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKiJgB3RjTyRjzqTFmnTFmrTHmttDwlsaYecaYDaFui+pPLhERefFTQz8GYKKInAZgEIBfG2NOAzAJwHwR6Qlgfug7ERHVkJjrxPgDAAAPP0lEQVQBXUR2iMiyUP8BAHkAOgAYBeDl0GgvA7i8uhJJRESxxdWGbozpCqA/gC8BtBGRHaGfdgJok9SUERFRXHwHdGNMEwDvALhdRPY7fxMRASAe0/3cGJNrjMktLCysUmKJiMibr4BujKkPDeavi8i7ocG7jDHtQr+3A1DgNq2IPCciOSKS07p162SkmYiIXPi5y8UAmAwgT0Qecfz0AYDxof7xAGYkP3lERORXuo9xhgC4HsBqY8yK0LA/AvgHgLeMMTcD2Arg6upJIhER+REzoIvIQgDG4+fhyU0OEREliv8UJSIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCggGdCKigGBAJyIKCAZ0IqKAYEAnIgoIBnQiooBgQCciCoiYAd0Y86IxpsAYs8YxrKUxZp4xZkOo26J6k0lERLH4qaFPAXBhxLBJAOaLSE8A80PfiYioBsUM6CLyOYA9EYNHAXg51P8ygMuTnC4iIopTom3obURkR6h/J4A2XiMaY35ujMk1xuQWFhYmuDgiIoqlyhdFRUQASJTfnxORHBHJad26dVUXR0REHhIN6LuMMe0AINQtSF6SiIgoEYkG9A8AjA/1jwcwIznJISKiRPm5bXEqgP8AOMUYk2+MuRnAPwCcb4zZAOC80HciIqpB6bFGEJFrPX4anuS0EBFRFfCfokREAcGATkQUEAzoREQBwYBORBQQDOhERAHBgE5EFBAM6EREAcGATkQUEAzoREQBwYBORBQQDOhERAHBgE5EFBAM6EREAcGATkQUEAzoREQBwYBORBQQDOhERAHBgE5EFBAM6EREAcGATkQUEAzoREQBwYBORBQQDOhERAHBgE5EFBAM6EREAcGATkQUEAzoREQBwYBORBQQDOhERAHBgE5EFBAM6EREAcGATkQUEAzoREQBwYBORBQQVQroxpgLjTHrjTEbjTGTkpUoIiKKX8IB3RiTBuApABcBOA3AtcaY05KVMCIiik96FaY9A8BGEdkMAMaYaQBGAViXjISFOXgQeO45YNYs4NZbgW++ASoqdPiiRcCgQUDLloAxwP79QMOGQJs2wOrVwNGjQFERcPXVwIIFQF4e8NOfAhMnAqefDjRqBJx4IvDjHwPt2wMNGgAnnaTTv/mmzuNHPwK6dwfuvlvHufJKoF07YO5c4NRTgfR0YOtW4OKLgYICYMMGneaUU4DCQl3+p58Cx47p8t58E8jMBM48E/j+e2DYMB3/66+BsjLg2WeBSy4BrrpK17l1a+DwYV3np58GJk3SNLdtCxw5Auzdq/Pp3h3YvVvXuX59YNMm7aalAS1aaDr27AGaN9fvHTsC/fvrMrp1A5Yv1+HNm+s82rbV9Ldsqet68cXAJ58AI0cCX3yhyxYBevXSZR08qPk6Z46m9fHHgQsuAM44Q/Nk507N8/XrNa35+cCBA8DgwcDUqcCFF+ryd+7UdV60CPjlL3WbbdwILF4MdOqky7r8cuCEE3S7rl8PTJkC/Oc/wO236zz69dNlzp4NfPcdUFwMjBqledCnjy63pETTOno00KWLjl+vHtC4sY5XVqbrP3Qo8N572v+LXwCffw7k5Oh2AvT3Vq30Y4ym8ZNPNH+3bgX27dN8GjoU2L5d592/v26rzEwts127an4uWQKce66Wk0aNNJ9XrwbKyzVte/YAO3YAJ5+s63jSSZpXu3fr9GvXatkcMEDL5YIFWj6mTtV1POccXVbz5sCuXcCKFbqtLroI+PBDYOBAnfazzzR/unTRdfr4Y6BJE2D+fN1/mjXTcTdt0v1x+XJg/Hhd1sGDwKFDmrZDh/S3+vWBvn11/PR0zcPTT9f1v+QS3WcbNdI0tG6tZa6sDFi1Srd9vXq6r7Vvr+XDGGDbNqBDB90Xzj1Xxyss1LxavFjLdIcOWj6WLdO8O/98zXtA1y09XcvQyScDf/wj0LSpbp+1a/X3Tz4BevTQz65dui8tWQJkZACdO+s2LyvT7Tx4MPDtt5oOQPef/v2Bv/0N+P3vNX+qmRGRxCY05ioAF4rILaHv1wMYKCITvKbJycmR3Nzc+BYkApx9NvDvfyeUTiKiWmHxYj0IJsAYs1REcmKNV5Uaut+E/BzAzwGgc+fOicwA+POfgSef1BrYWWfpEbSoSGsAFRV6ZF+9WmsQGRlaI8jK0qN15846HQA89ZTW1N56C1i5UmvhR44A48bpUfXkk/Uo/M03ds2qVy/9ftVVwNtv63zOPltrc5066VH5jTe0Zn7TTVoTOPVUXfZFF+lyFi3SGtGOHVqDycrS+ZaVaQ3yhBOA//ovrQV36aLzGz1a1+uNN3TctDTgvPN0uSedpPNftUprLgcPam2/f3/glVe0lrF2rdYeTjlFD4Zduuj6AloDW7xYp+3TR2uNQ4ZoTXHXLq29GqM1zA0bdNlbt2pNZt48PYs5cADIztb1y8kBpk3TbXLhhbpNcnJ0fbZv1/F69tT+Fi30bGXIEK05rV+vNafOnbUmbYymuUULTW+fProdjh3TvDrnHODFF3U9srKASy8FPvpIpz/pJN2OzZsDI0ZorfUvf9Fa2pYtOk1mJnDddZruHj10nq1aaW2qYUMdfuKJWpE49VRg+nRN68CBwObNmmc7dmjtb/VqPbP78EPdBqNGadkrL9dljRih/RUV9tnS3r2ax+PGaX506AA8+KCm/bzzNF+HDNEzskOHgJkzdf07ddJllZZq7bW8XLtLl2qts6REz/DWrNHaakEB0Lu3nt2Uluq2bdRI87igQPPzzDN1Xxg8WMv0fffpdho/Hnj3Xd3+bdtqPs6bp+nLyNBy++WXWs7OOENr/Pv3aznJzdXllJbqOO3b6xlP/fqaz5s2aXm2atcHDuiZX2Eh8MwzwGWXaV4OHQo8/7xOP3CgnqlkZ2u5OHAAmDFDy2FBgaZx2DDdtgsXal60aaPr1r+/5rm1/S29eum6FhRoLX3gQD1j7NhRl/nVV1oONm7U7ZeeruWjuFjX/8c/1rOfpk01HStWaF6sX69lduFCXc5vf6tnTGlp+ns1q0oNfTCAu0VkROj7HwBARO7zmiahGjoR0Q+c3xp6Ve5yWQKgpzGmmzGmAYAxAD6owvyIiKgKEm5yEZFjxpgJAOYASAPwooisTVrKiIgoLlVqQxeRjwB8lKS0EBFRFfCfokREAcGATkQUEAzoREQBwYBORBQQDOhERAGR8B+LElqYMYUAtiY4eSsAu5OYnGRhuuLDdMWH6YpPbU0XULW0dRGR1rFGSmlArwpjTK6ff0qlGtMVH6YrPkxXfGpruoDUpI1NLkREAcGATkQUEHUpoD9X0wnwwHTFh+mKD9MVn9qaLiAFaaszbehERBRdXaqhExFRFHUioNfUy6iNMZ2MMZ8aY9YZY9YaY24LDb/bGLPdGLMi9BnpmOYPoXSuN8aMqOb0bTHGrA6lITc0rKUxZp4xZkOo2yI03Bhj/i+UtlXGmOxqStMpjnxZYYzZb4y5vSbyzBjzojGmwBizxjEs7vwxxowPjb/BGFPl94h5pOtBY8zXoWW/Z4xpHhre1RhzxJFvzzimGRDa/htDaTfVkK64t1uy91ePdL3pSNMWY8yK0PBU5pdXfKi5MiYitfoDfTTvJgDdATQAsBLAaSladjsA2aH+pgC+gb4Q+24A/+My/mmh9DUE0C2U7rRqTN8WAK0ihj0AYFKofxKA+0P9IwF8DMAAGATgyxRtu50AutREngEYCiAbwJpE8wdASwCbQ90Wof4W1ZCuCwCkh/rvd6Srq3O8iPl8FUqrCaX9ompIV1zbrTr2V7d0Rfz+MIA/10B+ecWHGitjdaGGfvxl1CJSCsB6GXW1E5EdIrIs1H8AQB6ADlEmGQVgmogcFZFvAWyEpj+VRgF4OdT/MoDLHcNfEbUYQHNjTLtqTstwAJtEJNqfyaotz0TkcwB7XJYXT/6MADBPRPaIyF4A8wBcmOx0ichcETkW+roYQMdo8wil7UQRWSwaFV5xrEvS0hWF13ZL+v4aLV2hWvbVAKZGm0c15ZdXfKixMlYXAnoHANsc3/MRPahWC2NMVwD9AXwZGjQhdNr0onVKhdSnVQDMNcYsNfruVgBoIyI7Qv07AbSpobQB+hYr545WG/Is3vypiXy7CVqTs3Qzxiw3xnxmjDkrNKxDKC2pSFc82y3V+XUWgF0issExLOX5FREfaqyM1YWAXuOMMU0AvAPgdhHZD+BpACcDyAKwA3rKVxPOFJFsABcB+LUxZqjzx1BNpEZuYzL6WsLLAEwPDaoteXZcTeaPF2PMnQCOAXg9NGgHgM4i0h/A7wC8YYw5MYVJqnXbLcK1CK80pDy/XOLDcakuY3UhoG8H0MnxvWNoWEoYY+pDN9brIvIuAIjILhEpF5EKAM/DbiJIaVpFZHuoWwDgvVA6dllNKaFuQU2kDXqQWSYiu0JprBV5hvjzJ2XpM8bcAOASANeFAgFCTRpFof6l0PbpXqE0OJtlqiVdCWy3VOZXOoArALzpSG9K88stPqAGy1hdCOg19jLqUPvcZAB5IvKIY7iz7Xk0AOvq+wcAxhhjGhpjugHoCb0QUx1pa2yMaWr1Qy+qrQmlwbpKPh7ADEfaxoWutA8CsM9xWlgdwmpOtSHPHMuLJ3/mALjAGNMi1NxwQWhYUhljLgTwvwAuE5HDjuGtjTFpof7u0PzZHErbfmPMoFA5HedYl2SmK97tlsr99TwAX4vI8aaUVOaXV3xATZaxqlzlTdUHenX4G+jR9s4ULvdM6OnSKgArQp+RAF4FsDo0/AMA7RzT3BlK53pU8Sp6jLR1h95BsBLAWitfAGQCmA9gA4B/AWgZGm4APBVK22oAOdWYtsYAigA0cwxLeZ5BDyg7AJRB2yVvTiR/oG3aG0OfG6spXRuh7ahWOXsmNO6Voe27AsAyAJc65pMDDbCbADyJ0B8Fk5yuuLdbsvdXt3SFhk8BcGvEuKnML6/4UGNljP8UJSIKiLrQ5EJERD4woBMRBQQDOhFRQDCgExEFBAM6EVFAMKATEQUEAzoRUUAwoBMRBcT/B9+Nz4sdp++GAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe1d42c8450>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##################################################################################################\n",
    "def AddLeNetModel(model, data):\n",
    "    # Image size: 28 x 28 -> 24 x 24\n",
    "    conv1 = brew.conv(model, data, 'conv1', dim_in=1, dim_out=20, kernel=5, \\\n",
    "                weight_init=('ConstantFill', dict(value=1000.0)),\n",
    "                bias_init=('ConstantFill', dict(value=1000.0)))\n",
    "    # Image size: 24 x 24 -> 12 x 12\n",
    "    pool1 = model.net.MaxPool(conv1, 'pool1', kernel=2, stride=2)\n",
    "    # Image size: 12 x 12 -> 8 x 8\n",
    "    conv2 = brew.conv(model, pool1, 'conv2', dim_in=20, dim_out=50, kernel=5, \\\n",
    "                weight_init=('ConstantFill', dict(value=1000.0)),\n",
    "                bias_init=('ConstantFill', dict(value=1000.0)))\n",
    "    # Image size: 8 x 8 -> 4 x 4\n",
    "    pool2 = model.net.MaxPool(conv2, 'pool2', kernel=2, stride=2)\n",
    "    # 50 * 4 * 4 stands for dim_out from previous layer multiplied by the image size\n",
    "    fc3 = brew.fc(model, pool2, 'fc3', dim_in=50 * 4 * 4, dim_out=500, \\\n",
    "                weight_init=('ConstantFill', dict(value=1000.0)),\n",
    "                bias_init=('ConstantFill', dict(value=1000.0)))\n",
    "    fc3 = model.net.Relu(fc3, 'relu3')\n",
    "    pred = brew.fc(model, fc3, 'pred', 500, 10)\n",
    "    softmax = model.net.Softmax(pred, 'softmax')\n",
    "    return softmax\n",
    "###################################################################################################\n",
    "\n",
    "\n",
    "#TRAINING and TESTING\n",
    "train_model = model_helper.ModelHelper(name=\"mnist_train\", arg_scope=arg_scope)\n",
    "data, label = AddInput(\n",
    "    train_model, batch_size=64,\n",
    "    db=os.path.join(data_folder, 'mnist-train-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "\n",
    "softmax = AddModel(train_model, data)\n",
    "AddTrainingOperators(train_model, softmax, label)\n",
    "\n",
    "workspace.ResetWorkspace()\n",
    "workspace.RunNetOnce(train_model.param_init_net)\n",
    "workspace.CreateNet(train_model.net, overwrite=True)\n",
    "total_iters_c_h = 2000\n",
    "accuracy_c_h = np.zeros(total_iters_c_h)\n",
    "loss_c_h = np.zeros(total_iters_c_h)\n",
    "printmd('**Training status: Running**  Wait till the process is completed')\n",
    "for i in range(total_iters_c_h):\n",
    "    workspace.RunNet(train_model.net)\n",
    "    if(i%100 == 0):\n",
    "        print \"    Completed iterations:\", i, \", Total iterations to be completed:\", total_iters_c_h, \\\n",
    "        \", Current Loss = \", workspace.blobs['loss']\n",
    "    accuracy_c_h[i] = workspace.blobs['accuracy']\n",
    "    loss_c_h[i] = workspace.blobs['loss']\n",
    "#print \"Training completed\"\n",
    "printmd('**Training status: Completed**')\n",
    "\n",
    "#print \"Testing status: Running\"\n",
    "printmd('**Testing status: Running**  Wait till the process is completed')\n",
    "test_model = model_helper.ModelHelper(\n",
    "    name=\"mnist_test\", arg_scope=arg_scope, init_params=False)\n",
    "data, label = AddInput(\n",
    "    test_model, batch_size=100,\n",
    "    db=os.path.join(data_folder, 'mnist-test-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "softmax = AddModel(test_model, data)\n",
    "AddAccuracy(test_model, softmax, label)\n",
    "workspace.RunNetOnce(test_model.param_init_net)\n",
    "workspace.CreateNet(test_model.net, overwrite=True)\n",
    "test_iters_c_h = 1000\n",
    "test_accuracy_c_h = np.zeros(test_iters_c_h)\n",
    "for i in range(test_iters_c_h):\n",
    "    if(i%100 == 0):\n",
    "        print \"    Completed test set:\", i, \", Total sets to be tested:\", test_iters_c_h\n",
    "    workspace.RunNet(test_model.net.Proto().name)\n",
    "    test_accuracy_c_h[i] = workspace.FetchBlob('accuracy')\n",
    "#print \"Testing completed\"\n",
    "printmd('**Testing status: Completed**')\n",
    "# After the execution is done, let's plot the values.\n",
    "pyplot.plot(loss_c_h, 'b')\n",
    "pyplot.plot(accuracy_c_h, 'r')\n",
    "pyplot.legend(('Loss', 'Accuracy'), loc='upper right')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Effects of Initializers**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" cellpadding=\"3\" cellspacing=\"0\"  style=\"border:black; border-collapse:collapse;\"><tr><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Initializer&nbsp;Type</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Num&nbsp;iterations</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;LR</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;Loss</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Loss</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;Training-Acc(%)</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Training-Acc(%)</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Testing-Acc(%)</b></td></tr><tr><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">Constant&nbsp;-&nbsp;high</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">43.1735</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">41.7344</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">6.2500</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">9.3750</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.3200</td></tr></table>"
      ],
      "text/plain": [
       "<ipy_table.ipy_table.IpyTable at 0x7fe1cd247910>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "printmd('**Effects of Initializers**')\n",
    "\n",
    "initializers = [\n",
    "    ['Initializer Type', 'Num iterations', 'Init LR', 'Init Loss', 'Final Loss', 'Init Training-Acc(%)', \\\n",
    "     'Final Training-Acc(%)', 'Final Testing-Acc(%)'],\n",
    "    ['Constant - high', total_iters_c_h, base_lr, loss_c_h[0], loss_c_h[total_iters_c_h-1], accuracy_c_h[0]*100, \\\n",
    "     accuracy_c_h[total_iters_c_h-1]*100, np.mean(test_accuracy_c_h)*100],\n",
    "\n",
    "];\n",
    "make_table(initializers)\n",
    "apply_theme('basic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Constant initialization - Zero"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Training status: Running**  Wait till the process is completed"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Completed iterations: 0 , Total iterations to be completed: 2000 , Current Loss =  2.3025854\n",
      "    Completed iterations: 100 , Total iterations to be completed: 2000 , Current Loss =  2.302477\n",
      "    Completed iterations: 200 , Total iterations to be completed: 2000 , Current Loss =  2.3025482\n",
      "    Completed iterations: 300 , Total iterations to be completed: 2000 , Current Loss =  2.3018017\n",
      "    Completed iterations: 400 , Total iterations to be completed: 2000 , Current Loss =  2.3040066\n",
      "    Completed iterations: 500 , Total iterations to be completed: 2000 , Current Loss =  2.3035283\n",
      "    Completed iterations: 600 , Total iterations to be completed: 2000 , Current Loss =  2.2995036\n",
      "    Completed iterations: 700 , Total iterations to be completed: 2000 , Current Loss =  2.3060343\n",
      "    Completed iterations: 800 , Total iterations to be completed: 2000 , Current Loss =  2.3040578\n",
      "    Completed iterations: 900 , Total iterations to be completed: 2000 , Current Loss =  2.29848\n",
      "    Completed iterations: 1000 , Total iterations to be completed: 2000 , Current Loss =  2.3029509\n",
      "    Completed iterations: 1100 , Total iterations to be completed: 2000 , Current Loss =  2.301962\n",
      "    Completed iterations: 1200 , Total iterations to be completed: 2000 , Current Loss =  2.3042207\n",
      "    Completed iterations: 1300 , Total iterations to be completed: 2000 , Current Loss =  2.2985897\n",
      "    Completed iterations: 1400 , Total iterations to be completed: 2000 , Current Loss =  2.3015785\n",
      "    Completed iterations: 1500 , Total iterations to be completed: 2000 , Current Loss =  2.303531\n",
      "    Completed iterations: 1600 , Total iterations to be completed: 2000 , Current Loss =  2.296183\n",
      "    Completed iterations: 1700 , Total iterations to be completed: 2000 , Current Loss =  2.3042736\n",
      "    Completed iterations: 1800 , Total iterations to be completed: 2000 , Current Loss =  2.307296\n",
      "    Completed iterations: 1900 , Total iterations to be completed: 2000 , Current Loss =  2.2991672\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Training status: Completed**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Testing status: Running**  Wait till the process is completed"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Completed test set: 0 , Total sets to be tested: 1000\n",
      "    Completed test set: 100 , Total sets to be tested: 1000\n",
      "    Completed test set: 200 , Total sets to be tested: 1000\n",
      "    Completed test set: 300 , Total sets to be tested: 1000\n",
      "    Completed test set: 400 , Total sets to be tested: 1000\n",
      "    Completed test set: 500 , Total sets to be tested: 1000\n",
      "    Completed test set: 600 , Total sets to be tested: 1000\n",
      "    Completed test set: 700 , Total sets to be tested: 1000\n",
      "    Completed test set: 800 , Total sets to be tested: 1000\n",
      "    Completed test set: 900 , Total sets to be tested: 1000\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Testing status: Completed**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7ee1ccf7e9d0>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl4FGW2P/DvMYRFZQ1RkF2BUVDDkhGUwQ0Z0Qsigog6KjoOM+PK9Tdeueoog44OjnpVxisDwyJcDO7AiKiMiMjgliCK7CGCBNkEA4QAIcn5/XGq6OqkO92ddKdJ8f08Tz1dy1tVp96qPlVdXYuoKoiIyF9OSHYAREQUf0zuREQ+xORORORDTO5ERD7E5E5E5ENM7kREPsTkTkTkQ0zuREQ+xORORORDdZI14+bNm2v79u2TNXsiolopJyfnR1VNj1Quacm9ffv2yM7OTtbsiYhqJRHZHE05npYhIvIhJnciIh9icici8qGknXMnIv86cuQI8vPzcejQoWSHUmvVr18frVu3RmpqapXGZ3InorjLz89Hw4YN0b59e4hIssOpdVQVu3fvRn5+Pjp06FClafC0DBHF3aFDh5CWlsbEXkUigrS0tGr98mFyJ6KEYGKvnurWH5P7MejAgchlyr8dsbTU+pWWxjeW/fuBtWujL68aiM0bY1mZfZaWAgcPVhzH/SwqCvTfvbvi9PfuBfLzo19OVWDXror9jhyxeXnjcfu7y1BWFrwM7nB3edwYSkqA778PlPvpp8B4BQXWfuAAsH17cEzeZQUC0wYq1lNJCVBcDBw6ZJ/udIqKgG3bbHhJCVBYGDzN0lKrs8JCi2vfPmDlShu2bBmwdavF+P33NjyU/PyK21tuLrBhQ2B5P/nE1tfXXwM//hh6Oq5Q24hb367iYquPsrLA8rrrzC1bXBxYJwUF9llaap8HDgRPu7TU6set00OHgtd1aSlw+LD198Zw+LCNE+ptpMXFgTpTteV3twvvsrjz8C53Tah159wXLwbGjq3Y37uTc1di48b2JWrd2lb2kiVA8+ZAvXr2Reve3TaW1FTbKM88M3QiS0+vmCBOOw344Qdrb9TINhR3w2nSxL5QfD0tHa8WLIjuICWRLrzwZCxZUhi5YBx991105c44A2jaNLGx1LrkHkr5PWJhoe1x69WzPWh+PnDyyTbsxx+BE0+0JL9xI9CgQSBJhztCrVevYj93HMCOhup4arKgoHrLQ0T+tm8fk3sFF19sR+81TTX410FlZbw7Grc73Ljuz7cTQpwgUw3+6dmgge1E3J+RKSk23QMH7NdH3br2E71BA/v14P6krFPHdm6rVwN79thG1bWr/SQ/+WQ72sjIsHnUrWs7uTZtbKeY7jzBYtcuO9ooKbEymzYBp59uv1CKiwPliostrkOHbL716gE7dtjOUBXo1s3ms2+flU9PB9atA1q1CuyAi4psOu6pibw8oFkz2yEDQIsW9unWQfn6z862HfiOHTavU08FzjrLym7cCHTsaKcmzj7blqe01GLo2tV2zKWlVp+LF1t9XX21xduokcX42WfA5s3AuHHAK69YPGvXAr16BX6qT5sG3HSTLd+yZcDll9upj8aNbb3Mng00bAgMGGCfH39sy7B/vy1rejrQoUNgOVJSbL3l5FgdNmsGvPkm0KmTbUM9eth0S0qszk84wQ5kGjYE3nsvsAxbtgCnnGJ1/P77tk088YTNZ+5cW4aWLe2Uy/r1wLnn2vZVUAAsX27rdN8+2z4GDbJ2Vaunli0tpmHDbD4tWwL161tdpqTYuGVltk4bNrTp5uXZdlRUZHV75IiVq1PHxlW1uq5f39bjTz/ZsNRU61dSYtN2T8e437cGDawO2ra19oMH7TuxZcsm3H77bfjxxx9xyinpmDZtGtq2bYtXXnkdf/7znwCkoHHjxliwYAlWrVqFe++9FUeOFOPIkTI899ybuOCCTkcP5ESs2bfP1uvBgxYTYN+LRo0stv37LeaTTrL2wkL75Z9ookk6d5CZmal8tgyRP61ZswZnnXUWAGD0aGDFivhOv1s34LnnKi9z8skno7DcnxCDBg3CsGHDcMstt2Dq1KmYN28e5syZg3POOQfvvfceWrVqhYKCAjRp0gR33303evfujRtvvBHFxcUoLS1FgwYN4rsgEXjr0SUiOaqaGWlc/qFKRMeNTz/9FDfccAMA4KabbsLSpUsBAH369MHIkSMxefJklDr/lJ9//vl44oknMH78eGzevLnGE3t11brTMkRUu0Q6wj4WTJw4EZ9//jnmz5+Pnj17IicnBzfccAN69eqF+fPn48orr8Tf//53XHrppckONWo8ciei48YFF1yA2bNnAwBmzZqFvn37AgA2btyIXr16Ydy4cUhPT8eWLVuQl5eH008/Hffccw8GDx6Mb775Jpmhx4xH7kTkS0VFRWjduvXR7vvuuw8TJkzArbfeir/+9a9IT7c/VAHg/vvvx4YNG6Cq6NevHzIyMjB+/HjMnDkTqampaNGiBR588MFkLUqV8A9VIoq7UH8EUuz4hyoREQVhcici8iEmdyIiH2JyJyLyISZ3IiIfYnInIvIhJnci8q05c+ZARLA2lpcS+ASTOxH5VlZWFn7xi18gKysrYfMojfcbcuKEyZ2IfKmwsBBLly7FlClTjj5yAADGjx+Pc845BxkZGRgzZgwAIDc3F5dddhkyMjLQo0cPbNy4EYsXL8bAgQOPjnfXXXdh+vTpAID27dvjgQceQI8ePfD6669j8uTJ+PnPf46MjAwMHToURc5rtnbs2IEhQ4YgIyMDGRkZWLZsGR555BE853ngzkMPPYTnn38+7svPxw8QUWIl6Zm/c+fOxYABA9C5c2ekpaUhJycHO3fuxNy5c/H555/jxBNPxJ49ewAAN954I8aMGYMhQ4bg0KFDKCsrw5YtWyqdflpaGpYvXw4A2L17N37zm98AAB5++GFMmTIFd999N+655x5cdNFFePvtt1FaWorCwkKcdtppuOaaazB69GiUlZVh9uzZ+OKLL+JQKcGY3InIl7KysnDvvfcCAEaMGIGsrCyoKm699VaceOKJAIBmzZph//792Lp1K4YMGQIAqO++cSOC66677mj7t99+i4cffhgFBQUoLCzE5ZdfDgBYtGgRZsyYAQBISbEXgTRu3BhpaWn46quvsGPHDnTv3h1paWlxW24XkzsRJVYSnvm7Z88eLFq0CCtXroSIoLS0FCKCa6+9Nupp1KlTB2WeN10f8r49G8BJJ510tH3kyJGYM2cOMjIyMH36dCyO8Lq422+/HdOnT8f27dtx2223RR1TLHjOnYh854033sBNN92EzZs3Y9OmTdiyZQs6dOiAxo0bY9q0aUfPie/ZswcNGzZE69atMWfOHADA4cOHUVRUhHbt2mH16tU4fPgwCgoK8OGHH4ad3/79+9GyZUscOXIEs2bNOtq/X79+eOmllwDYH6979+4FAAwZMgTvvfcevvzyy6NH+fHG5E5EvpOVlXX0NItr6NCh2LZtG6666ipkZmaiW7duePrppwEAM2fOxAsvvIBzzz0XF1xwAbZv3442bdpg+PDhOPvsszF8+HB079497Pwee+wx9OrVC3369MGZZ555tP/zzz+Pjz76COeccw569uyJ1atXAwDq1q2LSy65BMOHD0eK90XAcRTxkb8i0gbADACnAlAAk1T1+XJlBMDzAK4EUARgpKour2y6fOQvkX/xkb+VKysrO3qlTadOncKWS/Qjf0sA/D9V7QKgN4A7RaRLuTJXAOjkNKMAvBTFdImIjjurV69Gx44d0a9fv0oTe3VF/ENVVbcB2Oa07xeRNQBaAVjtKTYYwAy1nwGfiUgTEWnpjEtERI4uXbogLy8v4fOJ6Zy7iLQH0B3A5+UGtQLgvSg03+lHRMepZL3lzS+qW39RJ3cRORnAmwBGq+q+qsxMREaJSLaIZO/atasqkyCiWqB+/frYvXs3E3wVqSp2794d9TX3oUR1nbuIpMIS+yxVfStEka0A2ni6Wzv9gqjqJACTAPtDNeZoiahWaN26NfLz88GDuKqrX79+0Au+YxUxuTtXwkwBsEZVnw1TbB6Au0RkNoBeAPbyfDvR8Ss1NRUdOnRIdhjHtWiO3PsAuAnAShFxHxDxIIC2AKCqEwG8C7sMMhd2KeSt8Q+ViIiiFc3VMksBSIQyCuDOeAVFRETVwztUiYh8iMmdiMiHmNyJiHyIyZ2IyIeY3ImIfIjJnYjIh5jciYh8iMmdiMiHmNyJiHyIyZ2IyIeY3ImIfIjJnYjIh5jciYh8iMmdiMiHmNyJiHyIyZ2IyIeY3ImIfIjJnYjIh5jciYh8iMmdiMiHmNyJiHyIyZ2IyIeY3ImIfIjJnYjIh5jciYh8iMmdiMiHmNyJiHyIyZ2IyIeY3ImIfIjJnYjIh5jciYh8iMmdiMiHmNyJiHyIyZ2IyIciJncRmSoiO0Xk2zDDLxaRvSKywmkeiX+YREQUizpRlJkO4G8AZlRS5hNVHRiXiIiIqNoiHrmr6hIAe2ogFiIiipN4nXM/X0S+FpEFItI1TtMkIqIqiua0TCTLAbRT1UIRuRLAHACdQhUUkVEARgFA27Zt4zBrIiIKpdpH7qq6T1ULnfZ3AaSKSPMwZSepaqaqZqanp1d31kREFEa1k7uItBARcdrPc6a5u7rTJSKiqot4WkZEsgBcDKC5iOQDeBRAKgCo6kQAwwD8XkRKABwEMEJVNWERExFRRBGTu6peH2H432CXShIR0TGCd6gSEfkQkzsRkQ8xuRMR+RCTOxGRDzG5ExH5EJM7EZEPMbkTEfkQkzsRkQ8xuRMR+RCTOxGRDzG5ExH5EJM7EZEPMbkTEfkQkzsRkQ8xuRMR+RCTOxGRDzG5ExH5EJM7EZEPMbkTEfkQkzsRkQ8xuRMR+RCTOxGRDzG5ExH5EJM7EZEPMbkTEfkQkzsRkQ8xuRMR+RCTOxGRDzG5ExH5EJM7EZEPMbkTEfkQkzsRkQ8xuRMR+RCTOxGRD0VM7iIyVUR2isi3YYaLiLwgIrki8o2I9Ih/mEREFItojtynAxhQyfArAHRymlEAXqp+WEREVB0Rk7uqLgGwp5IigwHMUPMZgCYi0jJeARIRUezicc69FYAtnu58px8RESVJjf6hKiKjRCRbRLJ37dpVk7MmIjquxCO5bwXQxtPd2ulXgapOUtVMVc1MT0+Pw6yJiCiUeCT3eQBudq6a6Q1gr6pui8N0iYioiupEKiAiWQAuBtBcRPIBPAogFQBUdSKAdwFcCSAXQBGAWxMVLBERRSdiclfV6yMMVwB3xi0iIiKqNt6hSkTkQ0zuREQ+xORORORDTO5ERD7E5E5E5ENM7kREPsTkTkTkQ0zuREQ+xORORORDTO5ERD7E5E5E5ENM7kREPsTkTkTkQ0zuREQ+xORORORDTO5ERD7E5E5E5ENM7kREPsTkTkTkQ0zuREQ+xORORORDTO5ERD7E5E5E5ENM7kREPsTkTkTkQ0zuREQ+xORORORDTO5ERD7E5E5E5ENM7kREPsTkTkTkQ0zuREQ+xORORORDTO5ERD7E5E5E5ENRJXcRGSAi60QkV0TGhBg+UkR2icgKp7k9/qESEVG06kQqICIpAF4E0B9APoAvRWSeqq4uV/RVVb0rATESEVGMojlyPw9ArqrmqWoxgNkABic2LCIiqo5oknsrAFs83flOv/KGisg3IvKGiLQJNSERGSUi2SKSvWvXriqES0RE0YjXH6r/BNBeVc8FsBDAy6EKqeokVc1U1cz09PQ4zZqIiMqLJrlvBeA9Em/t9DtKVXer6mGn8x8AesYnPCIiqopokvuXADqJSAcRqQtgBIB53gIi0tLTeRWANfELkYiIYhXxahlVLRGRuwC8DyAFwFRVXSUi4wBkq+o8APeIyFUASgDsATAygTETEVEEoqpJmXFmZqZmZ2cnZd5ERLWViOSoamakcrxDlYjIh5jciYh8iMmdiMiHmNyJiHyIyZ2IyIeY3ImIfIjJnYjIh5jciYh8iMmdiMiHmNyJiHyIyZ2IyIeY3ImIfIjJnYjIh5jciYh8iMmdiMiHmNyJiHyIyZ2I4q+kBHjgAeDHH5MdyXGLyT1aGzYAe/cmO4rq27SJX7ho5ecDO3aEH758OVBWVnPx1CYLFgBPPQXcfXeyIzluMblHq3Nn4MILkx1F9XXoYA1F1qYN0KJF6GH//jfQsyfwzDM1G1Nt4b6+s7AwuXEcx/yf3O+/H+jUqWrjjh8PNG0a6P7mm/jEFKvduwER4N134zO9WL5wl1wC3HZbfOZbFX/+M5CWlrz5h/Pdd/a5YkXVxi8utnU6dap1168PPPdcxXJffmnl8vKqNp9Zs4CUFODwYeC112xaNZFw69Sxz5KSxM/rWLNnj9Xz/PlJDaN2JveiIuCRR2yDjeTpp4Hc3KrNZ8wYoKCgYv9nngE2b7b2CROA9eut/a23gEWLgMces/kuX15x3I8/Bt58M9C9cCEwb174GObMATKdd+E+9ZQdEf3lL8CWLYEyb70FLF4cevxPPrEvdbQKCoCxY4HSUutevBiYNi386Yd33rGf4Iny8MP2ZfnXv6If5913gffes/YJE+yUWizeeQf44IPww//2N2DdOmsXCV0mKwv49NPgfocP23b72WeBI/4HHrC6PXwY+M//tG3uwQeBQ4ds+JQp9ukuT6zGjbPp5+XZvIHgbWflSmDSpNDjvvaabT9u7PfdF1juSFJS7NPdjlw7dwKPP15xe8rJAV5+Obpp5+YCzz8fuZyqHRzs3BnddCuzaJEd5Ljb4f/9n+14Q3EPAp96qvrzrQ5VTUrTs2dPrbKxY1UB1T/8QXXv3srL2ipWPXhQdePGisM3blQtKlLNzbUyZWWqq1YFj1taGmjfutU+u3ZVPXIk0N/b7m3CxZOfr7pnT3C5NWtUS0qs/eBBi8k7rQsvVM3Jsfb+/StOs6wseF7e8VVtucrHVVam+u23ge7bbrPhb70VPO1Jkyqv30g2brRlcu3Yobp4cWBYUVHl0wdUv/tOtbAw8rzc8ocO2ecpp0QeJ9w8AdXNm1W3bLFtraQkeNivfmXj/PCDrU9Vq89Q9fLssxWnXaeO6qZNFfvffLOtm9/+1rrHjFHdty9y7Fu2qBYUBLrPP9/GX7pU9YwzrH3duorLqmrb344d1l5+W1m61NqHDLFyqqrbt6vu2hU8/1WrLO5//cvKn3GG9XfX3eDB1v/VV62+PvlEde3ayrcjd5rud7N1ayu7f3/ldfHPf1q5vn1VFy6073Ek69apFhcHckD5enLrz23/+mvVDRuCy378sQ3r00c1Ly/8tl1FALI1ihxbO5P7f/1XoHI7doxUE9YMGxb4wrvcpP3LXwY23BkzrP3ddysmCfeLDqi2amXJyu1/772xJXdAtWHDQPv69fb50ENW7tprK06rb1/V+fMDMZef5j/+Eeh34EDwuO543uVQVZ02zbo/+MC6hw+37qysivGGSi7RJHd3x3f11RXHe//9isPC1Zf7hYnELVtUZJ+pqZHHqWyebnPGGcHbAmD15Y7ToIHq22+HX/9/+lP4aYdqJkwIJHdA9Zxzoou9ZctA98UXW79Fi1Tbt7d29yBn5crAtMvK7LN790By8i7DwoXB/bzjutyEPnmy6kcfBQ93112/fpUvc3mLF1v/iROt8Zb17sSiWY+PPVZ5+fx8K3fqqfY5b174aZVvvLG4O8Leve1zwIDK5xsjfyf3++8PvUGMHavaooUdDa1f79aENfXq2efFF6uOGhVYAd6mXr3QSdq7QXbtap+tWtmRg9v/3HNDr3R3BUdqPv00unJuM3Cgd21bc8cdtgFfdJHVQ2XjZ2fbuHfcYd3PPmvdbnKvW9fq0TvOzp1W5t//tu7Ro0N/Kd96S7Vp08ARy4ABgXLXXac6dGig+5lngtu9y3PVVaFjL/8LRVV1yRLVlBQ7cg2XNJYssfaLLlJt0kR17lzbIbplUlNVu3SpvN68v7bcxruTb9o09Hi5uaqPPBLbOgZUb7opuHvHjkCSjrVxY8vMrDjszTcD7a++WrXpu82ZZ1Z93OHDbZld//u/4cu62+Po0YEEOnSo/erxbkdu4/3OhPLkkxXHeeWV0NMK1dx2m5Vdtsy6e/YMDDvhBPvV7/56ysurPJZK+Du5n356cKUGljq4+dWvwq+Idu2qtwEDtpOo7jTcplmz2McZPDhwGqW6TYsWqlOmWIIMV6ayo64//tF2jt4E9vvfq7ZpU/l8n3oquPuSSyLHet11wduDe9RZWbN8eXzq6dJLK/abOze6cRs0qP78H3ssPstRvhEJtJ93XmLmEUszfLjq44/br5BwZebMCfw6A1R/85tAe/nTZ27z+OOqTzxhv8D++EfVBx6wU0OPPhq6fNOmsX3H2rdXPeus0MNSUwPt7sFUFfg7uZevtPXrVd95J/kbJJvYm5Ejqzbek0/aKZD161Wffjr5y8EmOc3114fu/7vfVT5ep07JjXvChGqkv+iSu1jZmpeZmanZ2dlVGzncFQpERLXBxInAb39bpVFFJEdVMyOVq32XQs6dm+wIiIiq5403Ej6L2pfc3et/iYhqqzvuSPgsal9y57M8qu+kk5IdAdHxq1s3YMiQhM/GX8m9YcPYpnXKKdWLJZJhwwJ3GIby+ONVn3aHDvbXTKxSUhLz4LDyd3SOGxd5nLPOin8clWne3OrsWNy5zZqV7Agiu+KK6k+jY8dA++9+V/3pAcDvfx+f6Xj9z/9U7LdmTXym/dVX8ZlOBLUvudeta5/DhgFXXx3o36ePPfHQa9684EcAeG/hHjAA+OEHu7167Vrg7bdD/8HRqlX4WDZsAPr2tfZbbqn4/I82bYATylXxhx8G2rt2DT/tSNzbwkNJT7fP5s2BVauCh3XqZM8x8Vq4MLi7Y0d7MFb5W82/+CKwQ7rqKvts1AhYvRro3z+47EMP2WMWpk+vWAcffWTjfP01cOqp4ZfD6513Kj5Gok+f6MZ1ubfZb94MnH12bONG8wiLjz+ObZrepHT99eHLTZ5sj1RYujTQL9TOs0GD4EdN/PBDbPF4/fOftu0sWhToN38+kJ0N3HNP5PG3b7fHHHgftpeRYY8ZcPXta9vBk08G+p1xRnC9zJwJfP998LRffz14OZ9+Onwc7dpFjrW8L74AmjQJ7rdrF3DmmTasvGefDbT37m2PDHnxRes+99zgslu3xh5PVUVzSU0imipfCnnkiF2TWlAQfBPRxIk23Hs9s+vDD+0mDVXVl15Sveaa0HdbFhfbddqffVbxUqsRI1RfeCHQ/+WXbZyDB+16WffWeve67ptvtvhefjn4EijVQHtJierPfmbXdl95pcXlvUX9vPPsGurvv7drbfv3r7hsbnfTpna36Rtv2M0STZrYp6rql18Gyn3/vfWbOtW6Fy2y7qws1V697Drg/PyK03dveioqsuU9cMCuEV6xIlD2H/9QveeewN2trvL3G3gfIbB2rep996m+/nroS8batbNLHl0LFgSG3Xuv6oknBpc/7zy7g3ngwMANWW4zZ05gOiNGWL/LLgsMd+vEbc4+267V//hjG2fVKtUXX7Q7Td0bfRo0sJjmz7cykydXXAbvtfxz51oc7jbjXZ/Llqm2bav6618Hj//++xXXx4QJ9tmli+qgQbZ9uI8V+Pvf7Q5Sb/mBA21drV1r444ebduM+zgAt+nQwT69t9Q/91zgpkDV4MdxhGpatAiU/eknu7nwscfscQXemI4cse7i4kC/nTuD7wJ2zZxp68rdDr33NninOWVKoH3cOLsb1xvbpZdaLCNH2vbrft+9N0u5MXmvfffeOPfaa8HTLCtT7dbN2t17MA4ftnW8b5/FMGhQ8PZXDfD1de5eixbZ7cLuxqxaccOoCnca7t2YblJs0sS+rOHMnGnl3WTv3jHZtatqjx7RxbdmjQ3/+uvg/uU3aFXbIURze/PIkaoXXBC5XHnxqEu3Dh980G50CveMDzdh9u9vO7tIMeXkBJL76tX2uWxZoJz7HCD3tv9t2wLD3DuUt2yxRz1cfrn1b9w4MP0ZM8LHUFJid/C6BxUud9117mx3/7rbebh67N3b1k15Dz8cnPBc/fvbAUdeng374ovwMarajrNLl/DD3XgB2zG/8oq1HzhQ+XTdG50efdR2eHXrBm7Smj698nEHDLA7Sb06d7adsuuyyyrerFZe//6qN95o7QMH2h3NqoHlOXzYuv/jP6Lbjm+80e5g93r6adW0tIplhwyx6TVqZN1ZWdbtPo4igeKa3AEMALAOQC6AMSGG1wPwqjP8cwDtI00zbsk9FPdRA9URj6QWTqtWiZt2vPXpc2zHOmiQxed9ZlB1ub+e3CP2WLg7lFGjgvu7d1pS4oU62gbssROJ4j4+5C9/Sdw8HNEm9zqRTtuISAqAFwH0B5AP4EsRmaeqqz3Ffg3gJ1XtKCIjAIwHcF18ThxVQX4+cOBA9aaxdWvibpZauTL0o4SPRQsWANu2JTuK8GbPtnPo9erFb5qjRwMXXwx07x77uKedZuu3c+fg/qtWAT/9FJfwKEre7+/atZX/f1ZdvXvb/3sZGYmbR4wi3qEqIucDGKuqlzvd/w0Aqvqkp8z7TplPRaQOgO0A0rWSiVfrDlUionBycuyZ+XfemexIEiLaO1QjHrkDaAXA83R/5APoFa6MqpaIyF4AaQCCrrkTkVEARgFA27Zto5g1EVGMeva05jhXo5dCquokVc1U1cx093I9IiKKu2iS+1YAbTzdrZ1+Ics4p2UaA9gdjwCJiCh20ST3LwF0EpEOIlIXwAgA5V/6OQ/ALU77MACLKjvfTkREiRXxnLtzDv0uAO8DSAEwVVVXicg42CU58wBMATBTRHIB7IHtAIiIKEmi+UMVqvougHfL9XvE034IwLXxDY2IiKqq9j1bhoiIImJyJyLyISZ3IiIfSto7VEVkF4DNVRy9OcrdIHWMOFbjAo7d2BhXbBhXbPwYVztVjXijUNKSe3WISHY0t9/WtGM1LuDYjY1xxYZxxeZ4jounZYiIfIjJnYjIh2prcp+U7ADCOFbjAo7d2BhXbBhXbI7buGrlOXciIqpcbT1yJyKiStSloJ49AAAE4klEQVS65C4iA0RknYjkisiYGp53GxH5SERWi8gqEbnX6T9WRLaKyAqnudIzzn87sa4TkcsTGNsmEVnpzD/b6ddMRBaKyAbns6nTX0TkBSeub0SkR4Ji+pmnTlaIyD4RGZ2M+hKRqSKyU0S+9fSLuX5E5Ban/AYRuSXUvOIQ119FZK0z77dFpInTv72IHPTU20TPOD2d9Z/rxF6t14iFiSvm9Rbv72uYuF71xLRJRFY4/WuyvsLlhuRtY9G8i+9YaWAPLtsI4HQAdQF8DaBLDc6/JYAeTntDAOsBdAEwFsAfQpTv4sRYD0AHJ/aUBMW2CUDzcv2egvPOWwBjAIx32q8EsACAAOgN4PMaWnfbAbRLRn0BuBBADwDfVrV+ADQDkOd8NnXamyYgrl8CqOO0j/fE1d5brtx0vnBiFSf2KxIQV0zrLRHf11BxlRv+DIBHklBf4XJD0rax2nbkfh6AXFXNU9ViALMBDK6pmavqNlVd7rTvB7AG9haqcAYDmK2qh1X1O9gLxM9LfKRB83/ZaX8ZwNWe/jPUfAagiYi0THAs/QBsVNXKblxLWH2p6hLYE0vLzy+W+rkcwEJV3aOqPwFYCHt5fFzjUtUPVLXE6fwM9g6FsJzYGqnqZ2oZYoZnWeIWVyXCrbe4f18ri8s5+h4OIKuyaSSovsLlhqRtY7UtuYd65V8C33obnoi0B9AdwOdOr7ucn1dT3Z9eqNl4FcAHIpIj9jpDADhVVd23W28HcGoS4nKNQPCXLtn1BcReP8mot9tgR3iuDiLylYh8LCJ9nX6tnFhqIq5Y1ltN11dfADtUdYOnX43XV7nckLRtrLYl92OCiJwM4E0Ao1V1H4CXAJwBoBuAbbCfhjXtF6raA8AVAO4UkQu9A50jlKRcGiX2kperALzu9DoW6itIMusnHBF5CEAJgFlOr20A2qpqdwD3AXhFRBrVYEjH3Hor53oEH0DUeH2FyA1H1fQ2VtuSezSv/EsoEUmFrbxZqvoWAKjqDlUtVdUyAJMROJVQY/Gq6lbncyeAt50YdrinW5zPnTUdl+MKAMtVdYcTY9LryxFr/dRYfCIyEsBAADc6SQHOaY/dTnsO7Hx2ZycG76mbhMRVhfVWk/VVB8A1AF71xFuj9RUqNyCJ21htS+7RvPIvYZxzelMArFHVZz39veerhwBw/8mfB2CEiNQTkQ4AOsH+yIl3XCeJSEO3HfaH3LcIfv3hLQDmeuK62fnHvjeAvZ6fjokQdESV7PryiLV+3gfwSxFp6pyS+KXTL65EZACA/wJwlaoWefqni0iK0346rH7ynNj2iUhvZxu92bMs8Ywr1vVWk9/XywCsVdWjp1tqsr7C5QYkcxurzj/EyWhg/zKvh+2FH6rhef8C9rPqGwArnOZKADMBrHT6zwPQ0jPOQ06s61DNf+Qriet02JUIXwNY5dYLgDQAHwLYAOBfAJo5/QXAi05cKwFkJrDOToK9LL2xp1+N1xds57INwBHYecxfV6V+YOfAc53m1gTFlQs77+puYxOdskOd9bsCwHIAgzzTyYQl240A/gbnBsU4xxXzeov39zVUXE7/6QB+V65sTdZXuNyQtG2Md6gSEflQbTstQ0REUWByJyLyISZ3IiIfYnInIvIhJnciIh9icici8iEmdyIiH2JyJyLyof8PyHb0ubs86BMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe1cd3d4f10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##################################################################################################\n",
    "def AddLeNetModel(model, data):\n",
    "    # Image size: 28 x 28 -> 24 x 24\n",
    "    conv1 = brew.conv(model, data, 'conv1', dim_in=1, dim_out=20, kernel=5, \\\n",
    "                weight_init=('ConstantFill', dict(value=0.)),\n",
    "                bias_init=('ConstantFill', dict(value=0.)))\n",
    "    # Image size: 24 x 24 -> 12 x 12\n",
    "    pool1 = model.net.MaxPool(conv1, 'pool1', kernel=2, stride=2)\n",
    "    # Image size: 12 x 12 -> 8 x 8\n",
    "    conv2 = brew.conv(model, pool1, 'conv2', dim_in=20, dim_out=50, kernel=5, \\\n",
    "                weight_init=('ConstantFill', dict(value=0.)),\n",
    "                bias_init=('ConstantFill', dict(value=0.)))\n",
    "    # Image size: 8 x 8 -> 4 x 4\n",
    "    pool2 = model.net.MaxPool(conv2, 'pool2', kernel=2, stride=2)\n",
    "    # 50 * 4 * 4 stands for dim_out from previous layer multiplied by the image size\n",
    "    fc3 = brew.fc(model, pool2, 'fc3', dim_in=50 * 4 * 4, dim_out=500, \\\n",
    "                weight_init=('ConstantFill', dict(value=0.)),\n",
    "                bias_init=('ConstantFill', dict(value=0.)))\n",
    "    fc3 = model.net.Relu(fc3, 'relu3')\n",
    "    pred = brew.fc(model, fc3, 'pred', 500, 10)\n",
    "    softmax = model.net.Softmax(pred, 'softmax')\n",
    "    return softmax\n",
    "###################################################################################################\n",
    "\n",
    "\n",
    "#TRAINING and TESTING\n",
    "train_model = model_helper.ModelHelper(name=\"mnist_train\", arg_scope=arg_scope)\n",
    "data, label = AddInput(\n",
    "    train_model, batch_size=64,\n",
    "    db=os.path.join(data_folder, 'mnist-train-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "\n",
    "softmax = AddModel(train_model, data)\n",
    "AddTrainingOperators(train_model, softmax, label)\n",
    "\n",
    "workspace.ResetWorkspace()\n",
    "workspace.RunNetOnce(train_model.param_init_net)\n",
    "workspace.CreateNet(train_model.net, overwrite=True)\n",
    "total_iters_c_m = 2000\n",
    "accuracy_c_m = np.zeros(total_iters_c_m)\n",
    "loss_c_m = np.zeros(total_iters_c_m)\n",
    "printmd('**Training status: Running**  Wait till the process is completed')\n",
    "for i in range(total_iters_c_m):\n",
    "    workspace.RunNet(train_model.net)\n",
    "    if(i%100 == 0):\n",
    "        print \"    Completed iterations:\", i, \", Total iterations to be completed:\", total_iters_c_m, \\\n",
    "        \", Current Loss = \", workspace.blobs['loss']\n",
    "    accuracy_c_m[i] = workspace.blobs['accuracy']\n",
    "    loss_c_m[i] = workspace.blobs['loss']\n",
    "#print \"Training completed\"\n",
    "printmd('**Training status: Completed**')\n",
    "\n",
    "#print \"Testing status: Running\"\n",
    "printmd('**Testing status: Running**  Wait till the process is completed')\n",
    "test_model = model_helper.ModelHelper(\n",
    "    name=\"mnist_test\", arg_scope=arg_scope, init_params=False)\n",
    "data, label = AddInput(\n",
    "    test_model, batch_size=100,\n",
    "    db=os.path.join(data_folder, 'mnist-test-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "softmax = AddModel(test_model, data)\n",
    "AddAccuracy(test_model, softmax, label)\n",
    "workspace.RunNetOnce(test_model.param_init_net)\n",
    "workspace.CreateNet(test_model.net, overwrite=True)\n",
    "test_iters_c_m = 1000\n",
    "test_accuracy_c_m = np.zeros(test_iters_c_m)\n",
    "for i in range(test_iters_c_m):\n",
    "    if(i%100 == 0):\n",
    "        print \"    Completed test set:\", i, \", Total sets to be tested:\", test_iters_c_m\n",
    "    workspace.RunNet(test_model.net.Proto().name)\n",
    "    test_accuracy_c_m[i] = workspace.FetchBlob('accuracy')\n",
    "#print \"Testing completed\"\n",
    "printmd('**Testing status: Completed**')\n",
    "# After the execution is done, let's plot the values.\n",
    "pyplot.plot(loss_c_m, 'b')\n",
    "pyplot.plot(accuracy_c_m, 'r')\n",
    "pyplot.legend(('Loss', 'Accuracy'), loc='upper right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Effects of Initializers**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" cellpadding=\"3\" cellspacing=\"0\"  style=\"border:black; border-collapse:collapse;\"><tr><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Initializer&nbsp;Type</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Num&nbsp;iterations</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;LR</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;Loss</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Loss</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;Training-Acc(%)</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Training-Acc(%)</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Testing-Acc(%)</b></td></tr><tr><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">Constant&nbsp;-&nbsp;high</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">43.1735</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">41.7344</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">6.2500</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">9.3750</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.3200</td></tr><tr><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">Constant&nbsp;-&nbsp;zero</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2.3026</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2.3023</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.9375</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.9375</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">11.3500</td></tr></table>"
      ],
      "text/plain": [
       "<ipy_table.ipy_table.IpyTable at 0x7fe1cd29aa10>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "printmd('**Effects of Initializers**')\n",
    "\n",
    "initializers = [\n",
    "    ['Initializer Type', 'Num iterations', 'Init LR', 'Init Loss', 'Final Loss', 'Init Training-Acc(%)', \\\n",
    "     'Final Training-Acc(%)', 'Final Testing-Acc(%)'],\n",
    "    ['Constant - high', total_iters_c_h, base_lr, loss_c_h[0], loss_c_h[total_iters_c_h-1], accuracy_c_h[0]*100, \\\n",
    "     accuracy_c_h[total_iters_c_h-1]*100, np.mean(test_accuracy_c_h)*100],\n",
    "    ['Constant - zero', total_iters_c_m, base_lr, loss_c_m[0], loss_c_m[total_iters_c_m-1], accuracy_c_m[0]*100, \\\n",
    "     accuracy_c_m[total_iters_c_m-1]*100, np.mean(test_accuracy_c_m)*100],\n",
    "\n",
    "];\n",
    "make_table(initializers)\n",
    "apply_theme('basic')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Xavier initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Training status: Running**  Wait till the process is completed"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Completed iterations: 0 , Total iterations to be completed: 2000 , Current Loss =  3.203992\n",
      "    Completed iterations: 100 , Total iterations to be completed: 2000 , Current Loss =  0.71429974\n",
      "    Completed iterations: 200 , Total iterations to be completed: 2000 , Current Loss =  0.4001926\n",
      "    Completed iterations: 300 , Total iterations to be completed: 2000 , Current Loss =  0.32719496\n",
      "    Completed iterations: 400 , Total iterations to be completed: 2000 , Current Loss =  0.2724344\n",
      "    Completed iterations: 500 , Total iterations to be completed: 2000 , Current Loss =  0.36996093\n",
      "    Completed iterations: 600 , Total iterations to be completed: 2000 , Current Loss =  0.18547262\n",
      "    Completed iterations: 700 , Total iterations to be completed: 2000 , Current Loss =  0.38056624\n",
      "    Completed iterations: 800 , Total iterations to be completed: 2000 , Current Loss =  0.3353026\n",
      "    Completed iterations: 900 , Total iterations to be completed: 2000 , Current Loss =  0.3089176\n",
      "    Completed iterations: 1000 , Total iterations to be completed: 2000 , Current Loss =  0.25842714\n",
      "    Completed iterations: 1100 , Total iterations to be completed: 2000 , Current Loss =  0.11632501\n",
      "    Completed iterations: 1200 , Total iterations to be completed: 2000 , Current Loss =  0.20493755\n",
      "    Completed iterations: 1300 , Total iterations to be completed: 2000 , Current Loss =  0.14789242\n",
      "    Completed iterations: 1400 , Total iterations to be completed: 2000 , Current Loss =  0.15220013\n",
      "    Completed iterations: 1500 , Total iterations to be completed: 2000 , Current Loss =  0.28468782\n",
      "    Completed iterations: 1600 , Total iterations to be completed: 2000 , Current Loss =  0.29060417\n",
      "    Completed iterations: 1700 , Total iterations to be completed: 2000 , Current Loss =  0.08672796\n",
      "    Completed iterations: 1800 , Total iterations to be completed: 2000 , Current Loss =  0.12618428\n",
      "    Completed iterations: 1900 , Total iterations to be completed: 2000 , Current Loss =  0.21128239\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Training status: Completed**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Testing status: Running**  Wait till the process is completed"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Completed test set: 0 , Total sets to be tested: 1000\n",
      "    Completed test set: 100 , Total sets to be tested: 1000\n",
      "    Completed test set: 200 , Total sets to be tested: 1000\n",
      "    Completed test set: 300 , Total sets to be tested: 1000\n",
      "    Completed test set: 400 , Total sets to be tested: 1000\n",
      "    Completed test set: 500 , Total sets to be tested: 1000\n",
      "    Completed test set: 600 , Total sets to be tested: 1000\n",
      "    Completed test set: 700 , Total sets to be tested: 1000\n",
      "    Completed test set: 800 , Total sets to be tested: 1000\n",
      "    Completed test set: 900 , Total sets to be tested: 1000\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Testing status: Completed**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7ee1ccf02510>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJztnXeYFUXWxt/DDEEQUYZZRZBkQgSGMCIKooKSVBBURP0Uw8qikhbXFbOy6oq6BnAXBAmKJEEliWIARJDgkHMUHBAGGMIwwMCE8/1Rt+nue7vv7Rv73sv5PU8/3V1dXXU6vV1dXXWKmBmCIAhCclHKbQMEQRCEyCPiLgiCkISIuAuCICQhIu6CIAhJiIi7IAhCEiLiLgiCkISIuAuCICQhIu6CIAhJiIi7IAhCEpLqVsZVqlThWrVquZW9IAhCQrJ8+fKDzJweKJ5r4l6rVi1kZWW5lb0gCEJCQkS7nMSTahlBEIQkRMRdEAQhCRFxFwRBSEJcq3MXBCF5KSwsxO7du1FQUOC2KQlLuXLlUL16dZQuXTqk/UXcBUGIOLt370bFihVRq1YtEJHb5iQczIzc3Fzs3r0btWvXDikNqZYRBCHiFBQUIC0tTYQ9RIgIaWlpYX35iLgLghAVRNjDI9zzl3Divm4d8PLLwP79blsiCIIQvyScuG/cCPzrX8CBA25bIghCPHPuuee6bYKrJJy4a18qJSXu2iEIghDPJJy4l/JYzOyuHYIgJB47d+5E69at0bBhQ7Rp0wZ//PEHAGDKlCmoX78+MjIy0KpVKwDA+vXr0axZMzRq1AgNGzbE1q1b3TQ9aBKuKaSU3AUhsejfH1i1KrJpNmoEfPBB8Pv16dMHPXr0QI8ePTB69Gj07dsX06ZNw6BBgzBnzhxUq1YNR44cAQAMHz4c/fr1wwMPPIDTp0+juLg4sgcRZaTkLgjCWcPixYtx//33AwAefPBBLFy4EADQokULPPzwwxg5cuQZEb/uuuvw5ptvYvDgwdi1axfOOecc1+wOhYQruWviLiV3QUgMQilhx5rhw4dj6dKl+Oabb9C0aVMsX74c999/P6699lp888036NixIz7++GO0bt3abVMdk3Ald6mWEQQhVK6//npMmjQJADB+/HjccMMNAIDt27fj2muvxaBBg5Ceno7s7Gzs2LEDderUQd++fdG5c2esWbPGTdODJmFL7lItIwiCP06cOIHq1aufWR8wYACGDh2KRx55BO+88w7S09MxZswYAMAzzzyDrVu3gpnRpk0bZGRkYPDgwRg3bhxKly6Niy66CM8//7xbhxISCSfuUnIXBMEJJTYiMXfuXJ+wr776yids4MCBGDhwYMTtihUBq2WIqBwRLSOi1US0nohes4hTlogmE9E2IlpKRLWiYSwgJXdBEAQnOKlzPwWgNTNnAGgEoD0RNfeK8xiAw8x8GYD3AQyOrJk6UnIXBEEITEBxZ0W+Z7W0Z/IuN3cG8KlneSqANhQlr0FSchcEQQiMo9YyRJRCRKsA7AfwAzMv9YpSDUA2ADBzEYCjANIiaahui5pLyV0QBMEeR+LOzMXM3AhAdQDNiKh+KJkRUU8iyiKirAMhev6SkrsgCEJggmrnzsxHAMwD0N5r0x4AlwAAEaUCqAQg12L/EcycycyZ6enpIRksJXdBEITAOGktk05E53uWzwFwK4BNXtFmAOjhWb4bwFzm6JStpeQuCIJTpk2bBiLCpk3ekpX8OCm5VwUwj4jWAPgNqs59FhENIqJOnjijAKQR0TYAAwBErXGouB8QBMEpEydORMuWLTFx4sSo5RGvDsWctJZZw8yNmbkhM9dn5kGe8JeZeYZnuYCZ72Hmy5i5GTPviJbBUi0jCIIT8vPzsXDhQowaNeqMywEAGDx4MBo0aICMjIwznZS2bduGW265BRkZGWjSpAm2b9+O+fPn4/bbbz+zX+/evTF27FgAQK1atfDss8+iSZMmmDJlCkaOHIlrrrkGGRkZuOuuu3DixAkAQE5ODrp06YKMjAxkZGTg119/xcsvv4wPDA53XnjhBXz44YcRP/6E66Eq1TKCkGC45PN3+vTpaN++Pa644gqkpaVh+fLl2L9/P6ZPn46lS5eifPnyOHToEADggQcewMCBA9GlSxcUFBSgpKQE2dnZftNPS0vDihUrAAC5ubl4/PHHAQAvvvgiRo0ahT59+qBv37648cYb8fXXX6O4uBj5+fm4+OKL0bVrV/Tv3x8lJSWYNGkSli1bFoGTYibhxF1K7oIgOGHixIno168fAKB79+6YOHEimBmPPPIIypcvDwCoXLkyjh07hj179qBLly4AgHLlyjlK/9577z2zvG7dOrz44os4cuQI8vPz0a5dOwDK1cFnn30GAEhJSUGlSpVQqVIlpKWlYeXKlcjJyUHjxo2Rlhb5luMJJ+5ScheEBMMFn7+HDh3C3LlzsXbtWhARiouLQUS45557HKeRmppq8k9TUFBg2l6hQoUzyw8//DCmTZuGjIwMjB07FvPnz/eb9l//+leMHTsW+/btw6OPPurYpmAQl7+CICQdU6dOxYMPPohdu3Zh586dyM7ORu3atVGpUiWMGTPmTJ34oUOHULFiRVSvXh3Tpk0DAJw6dQonTpxAzZo1sWHDBpw6dQpHjhzBTz/9ZJvfsWPHULVqVRQWFmL8+PFnwtu0aYNhw4YBUD9ejx49CgDo0qULvvvuO/z2229nSvmRJuHEXUrugiAEYuLEiWeqWTTuuusu7N27F506dUJmZiYaNWqEd999FwAwbtw4DBkyBA0bNsT111+Pffv24ZJLLkG3bt1Qv359dOvWDY0bN7bN71//+heuvfZatGjRAnXr1j0T/uGHH2LevHlo0KABmjZtig0bNgAAypQpg5tvvhndunVDSkpKFM4AQFFqjh6QzMxMzsrKCnq/FSuApk2BadOAzp2jYJggCGGzceNGXHXVVW6bEbeUlJScaWlz+eWX28azOo9EtJyZMwPlISV3QRCEGLJhwwZcdtllaNOmjV9hD5eE+6Eqde6CICQy9erVw44dUesKdAYpuQuCEBXcqvJNFsI9fwkr7lJyF4T4pVy5csjNzRWBDxFmRm5uruM291ZItYwgCBGnevXq2L17N0J17S2oF6RxgO9gSThxl2oZQYh/Spcujdq1a7ttxllNwlXLSMldEAQhMAkn7lJyFwRBCEzCibuU3AVBEAKTcOIuJXdBEITAJJy4S8ldEAQhMAkn7lJyFwRBCEzCibuU3AVBEAKTcOIuJXdBEITAJKy4S8ldEATBnoQTd6mWEQRBCExAcSeiS4hoHhFtIKL1RNTPIs5NRHSUiFZ5ppejY65UywiCIDjBiW+ZIgBPM/MKIqoIYDkR/cDMG7zi/cLMt0feRDNSchcEQQhMwJI7M+9l5hWe5WMANgKoFm3D7JCSuyAIQmCCqnMnoloAGgNYarH5OiJaTUTfEtHVEbDNxgY1l5K7IAiCPY5d/hLRuQC+BNCfmfO8Nq8AUJOZ84moI4BpAHwGBySingB6AkCNGjVCMlhK7oIgCIFxVHInotJQwj6emb/y3s7Mecyc71meDaA0EVWxiDeCmTOZOTM9PT0kg6XkLgiCEBgnrWUIwCgAG5n5PZs4F3nigYiaedLNjaShGlrJfe3aaKQuCIKQHDgpubcA8CCA1oamjh2JqBcR9fLEuRvAOiJaDWAIgO4cpcETtZL7mDHRSF0QBCE5CFjnzswLAVCAOB8B+ChSRvmjVMJ1uxIEQYg9CSeV5Pc1IwiCIAAJKO5SchcEQQhMwklluXJqfuut7tohCIIQzyScuANAw4ZA+fJuWyEIghC/JKS4p6QAxcVuWyEIghC/iLgLgiAkIY7dD8QTWVlqXlCg18ELgiAIOglZctfo08dtCwRBEOKThBb3Tz5x2wJBEIT4JKHFXRAEQbBGxF0QBCEJSUhxr1vXbQsEQRDim4QU92HD3LZAEAQhvklIca9dW1/evds9OwRBEOKVhBT3mjX15dWr3bNDEAQhXklIcRcEQRD8k/DiLv7dBUEQfEl4cf/lF7ctEARBiD8SXtzfesttCwRBEOKPhBd3QRAEwRcRd0EQhCRExF0QBCEJCSjuRHQJEc0jog1EtJ6I+lnEISIaQkTbiGgNETWJjrk6LVvqy8zRzk0QBCGxcFJyLwLwNDPXA9AcwFNEVM8rTgcAl3umngCi7iDgjTf05R9/jHZugiAIiUVAcWfmvcy8wrN8DMBGANW8onUG8BkrlgA4n4iqRtxaA61a6cs7dkQzJ0EQhMQjqDp3IqoFoDGApV6bqgHINqzvhu8LIGp8802schIEQUgMHIs7EZ0L4EsA/Zk5L5TMiKgnEWURUdaBAwdCScKSmTMjlpQgCEJS4Ejciag0lLCPZ+avLKLsAXCJYb26J8wEM49g5kxmzkxPTw/FXkEQBMEBTlrLEIBRADYy83s20WYAeMjTaqY5gKPMvDeCdloyenS0cxAEQUhMnJTcWwB4EEBrIlrlmToSUS8i6uWJMxvADgDbAIwE8GR0zDXzyCOxyEUQBCHxSA0UgZkXAvDre5GZGcBTkTIqGJo3B5YsAb7/Hmjb1g0LBEEQ4o+E76G6aZOaz5rlrh2CIAjxRMKLe7lyan7qlLt2CIIgxBMJL+7nnKPmBQXu2iEIghBPiLgLgiAkIUkj7idPumuHIAhCPJHw4q7VuYu4C4Ig6CS8uKd6GnOKuAuCIOgkvLhfdZWaL1rkrh2CIAjxRMKL+/vvu22BIAhC/JHw4l6uHNCtG3DuuW5bIgiCED8kvLgDQK1aQH4+8MorblsiCIIQHySFuJcvr+aDBrlrhyAIQryQFOJOBrdmJSXu2SEIghAvJIW4G5GeqoIgCEki7qkGx8XS3l0QBCFJxN04aIeIuyAIQpKIe9WqwJgxarmw0F1bBEEQ4oGkEHdAF/U6dYBjx9y1RRAEwW2SRtx379aXhw1zzw5BEIR4IGnEXXP9C8ioTIIgCEkj7oIgCIJO0oi7sfNSTo57dgiCIMQDAcWdiEYT0X4iWmez/SYiOkpEqzzTy5E3MzApKfryf//rhgWCIAjxg5OS+1gA7QPE+YWZG3kmVzy89O7tRq6CIAjxSUBxZ+YFAA7FwJawqFDBbQsEQRDih0jVuV9HRKuJ6FsiujpCaYZFUZHbFgiCILhHauAoAVkBoCYz5xNRRwDTAFxuFZGIegLoCQA1atSIQNb2nDpl9jkjCIJwNhF2yZ2Z85g537M8G0BpIqpiE3cEM2cyc2Z6enq4WfslLy+qyQuCIMQ1YYs7EV1EpDyqE1EzT5q54aYbLrfc4rYFgiAI7uGkKeREAIsBXElEu4noMSLqRUS9PFHuBrCOiFYDGAKgOzNz9Ey254EH9IE7xK+7IAhnMwFrpZn5vgDbPwLwUcQsCoPPP1dTxYpAhw5uWyMIguAeSdND1Uh+vnRkEgTh7CYpxV0QBOFsR8RdEAQhCUlqcXfnt64gCIL7JLW4i193QRDOVpJa3GW4PUEQzlaSWtylOaQgCGcrSS3uy5e7bYEgCII7JLW4C4IgnK0kvbhLvbsgCGcjSS/u48a5bYEgCELsSXpxf+op4KO48HwjCIIQO5JS3AcONK/36eOOHYIgCG6RlOL+738Dc+e6bYUgCIJ7JKW4A0BamtsWCIIguEfSivvJk25bIAiC4B5JK+516rhtgSAIgnskrbinpwMtWujrhYXu2SIIghBrklbcAaCU4egmTHDPDkEQhFiT1OJ+/Li+LANmC4JwNpHU4r5ihb784Yfu2SEIghBrklrcjWzc6LYFgiAIsSOguBPRaCLaT0TrbLYTEQ0hom1EtIaImkTeTEEQBCEYnJTcxwJo72d7BwCXe6aeAIaFb5YgCIIQDgHFnZkXADjkJ0pnAJ+xYgmA84moaqQMFARBEIInEnXu1QBkG9Z3e8LilqNHgfPPB+bNi3HGEycC991nDvv8c+Duu8NL9513gCeeCC+NSJOTAzRqBOzYEX5as2cDN94IlJT4j3fJJQARcPq0HrZ7twrTpmnTfPd75RXgmWfMYY8+CnzwgW/cZ58FXnopsM2vvQY8/XTgeIEYORK4/37g+eeBQYNU2IkTQLNmQIMGKp+tW4HGjYGDB/X9jh8HMjMjOxzZF18A995rv/0//wGefFIt33sv8MYbQP36wObNypalS0PPu2NHoF07/fn59Vd1PWvUAJo31499zhzgllt875XJk4EuXQLnk5en0lu3DliwAKhXT793Xn1V3RfeLFumrseJE6EfXzRg5oATgFoA1tlsmwWgpWH9JwCZNnF7AsgCkFWjRg2ONoB50pg/X623aqXWS0qY16yJujm6IQUFvmHVqzOvXx84jZMnmevWZf7xR980nLBzp4p7zz162K5dzJdfzlylCvMXXzhLx4q5c31P+gUXMLds6Rv3gw+YO3RgXruW+aKL1PEbj4mZ+dprzWlp6VevznzqlB5vxw5lu3feVtPFF6t9cnOZL7zQvO3KK9X8/PPN53TZMrVcpow5/OmnmZ94QrejY0frm27BArU8ebL5+LZvZ65Th/mPP/Sw+fOZ69VjPnGCefx46/S0G9h7+t//9HS6dVNh119v3nfKFOZrrmG+9VYV9u676tr/+CPz4MHm+8J4fmvU0NMZPVrf9sknzGlpZjs++8y8fvPNat6smdrnn/9kHjDAnMeLL+rxW7ZU9t17L/N776kH1Jje2rXWx79pE3OFCmp5/Xrm2rWZ27RhfvxxPU7Vqmperx7z5587u2ecTh07qnu6SRPmOXPUcf373yr/rl2Zn3lG2bRune85DgIAWexEtx1F8i/uHwO4z7C+GUDVQGk2bdo0rAN0dhLM02WXMW/bpq+3aKHivfeeWl+0KEYGHTxob+Qrr5j3KSpSYj5pklpftUrFq19fPRDt2un7HjoU2IZXXtHjZ2Ux33gjc6lSvnYUFamb8Y47mBs00MOPHlXpTJumRLagQL1wLr7Y/43PzLxkiVpu1co+3qRJ6vxUqxbZB8976tTJWTxmdY68w43Ce/w488cf2+9/6aXmsN9+UyJ0772+8cuWVfO2be1tuuce+20jRjC/9JL1tvr1nR+zRm6udZz+/cO/Bo0b2z8HyT4ZCydBEktxvw3AtwAIQHMAy5ykGQtxnzXL95zuvrotj8IjDDA3b67i3Xef2jbhs0Lm885jHjvWeSZvv81cs6Za/vln5pQU5v371fpPP+kZZ2fry23bMj/4IPPddwd+uA4fNm/r1cv+hpkzh3nvXrW8cCHzl18yly6txEdDK9kEmg4csA5/7jnfsK++Cpxe06bO8q1Y0fwCSvSpVCnzlwCgSnZu2+Vv6tJFlZarVmV+7DH37UnGKTs7VFnjiIk7gIkA9gIohKpPfwxALwC9PNsJwH8BbAew1q5KxnuKhbgzM2/daj6n2sKF2MvXXKPidO/OvAF1+ffr71fbzznHnMgrrzBXrqyWT55UcbQXgDHxCy5Q86++Cq/kuX596Pu++qr7N65MMiXD1LVr9NJ2UgVrQ0RL7tGYoiru9eox3347MzOfKlD1dQPxpudo1cltj9ncpAmragark9+/vxJrYz3yihX6slZat9r39tvdvzFlkkmm8Ca75zvYqW9f37AbbghZ3s5ucTdcnJLCojPrgNePGYD5++9Du2Dlypnzkilxp3HjmK+7jnnePP/xbrvNfVvjbWrd2n0bojE9+2zg5/tvf9OXp061j/fhh/qyViUXA3FPevcDVFJ8ZtnycNevDy3hgoLATfOSmQED3LZAp1Mn6/Du3YHsbNWkrVIl6zjTpgH/93+qad1NN9nn8d13QFFR2KYG5PzzgV69gM8+U00crZre+eOvf3UWr1Gj4G0DzM1MAeCnn4B//MP5/mPHAqVLh5b3uec6i/fDD8Dixf7jDByofJIYmxC3N/TV1PyFX3WVHjZ/PnDDDfr6Lbeo+XvvAV27AlOnAp9+CkyZYs6rRg19+a23nB1DJHDyBojGFJOS+/HjqkmZv7fva6+F/naPRJ3c3/8eu9JIaqr66Tt5cvhpjRgRO7uN0yWX+IYZr7l3uEbPnr7bR41SPw6NLF2qb9danbz2mtr27rv6tssuM6c1bBhzenrwx9Ovn778zDPW9/Ps2ar54eTJzIsXM2/ezLxyJfN335nTWrtWtV4yhn30EfM33/jmW1Rk3XTTe0pLY96zRzWXXblS2aM1GdXO8fHjavmWW/Twt982p9O4sWr6yayavgLquLzz+/13Fa41HzVOV1xhXv/6a//X3TtcO95du8znd9Ei1ULr+HH1BQeoBgnM5oYFzMzFxeqcrlyp7p1Zs9S5tNOgWbPMTTm1Zpy9e1tfawdAqmU8U58+wT9wsZy2bDGvn3eeb5yRIyOT16uv6ucoJ0d9GhqrInbuVG2FA6Xz8ceqKsNfnH/9yzr8hhuCs7lePfP68uW+cZiZd++2Dtd48kn/263un8JC9c9Fo6RE/SzX2szOmKHHPXiQ+f/+T1+3ax2kNWXVpmHD1Pyyy5hPnw7+Xtfu765ddRuN6c+a5ftMaMd+7Bjzr78qodq1i/mXX5iHD1fbteqW777zzfPYMfViNLb4WLKEOS9PT//bb9W8TBnm6dPNzXT/8he17c8/Vcuy6dOZhw5VLb2sroM2ebf0MuanTUabduwwP195eep4/eF9zY12BEN2NvPGjdZp/Pyzua9LkIi4x3IytjW3mjZvtt9mbCIJMO/b5xvH+4ENdRo82P58GesA/aUxfbqyp6RElQhXr1adamrVUtu7dVMCZpfOww+reYMGzDNn6m3fvaeff1ZpZ2SYwzds8I2rcfCgKnEBqqmpEU0EH3pIzYns75+dO80di+wwlpKPHFF5AqpUph1XRoYSMS3e/v1m2+fMUXNjB6Rg0Nr7jhyph2lpV6+uhy1ebH3OvCkpUeewuDi0jh/G9LdsUfezN1qHs5wc/2n16GG2+c47fY8hK0tfP3AgsE2hoH1phMPu3eplEwFE3GM5vfGG/bZbb1U27dqlSg0NG5q3a5+0duuayGzapNLQwleuVNu09bvu0ktbb71lbYtVKUxL+9gxfV3rZLRli+8PMzvu9zQj/fxz6+tw9dX69k2bVCnK3/XSqksGDFDr5cqxSdzLlVPCYSXCa9eqJqtGPv1U7Td7thJvrS9COBTpP+v52DH9xTFmjOqoBKgSPDPzAw+o9bw89SL49VdVBcGsBMq7eigYvPffvFmJ+YkTepjWhBdQvVajxZ49avKHdq8Y7zkrTp9W9/mhQ6pKqqREHavWy1Qj0L0Zrrjn5upVSnGAiHskpoMHrcO9Pw8/+MA3TlaW+rwzPmDMzDfdpLanpuo33J49+n6nT/sXU0B1m9fQ6oG//VZv475lixKO7dvV+k03qeoMp+Tl6cJz/LgqcXjn642VuOfk6F3RJ09WomuFdqxz56pOTMbjLixULwPt38Tevep85eY6Px5mJQxhdvu2TFOz/cQJs7hrL+LnnlNxCwrUdXGTrVtVVUF+vrt2nDypbAmVw4fN1S+A3t3cinDFPc44e8X9iy9sxXod6tlus5zYc4qaNzeH161rXh8wQD3M69fr3fDteqC9+abavmaN+lTTMJZY/Yl7To7uAkBDe1CKinxLGGvX+r5gQsEqXyPaD9asLN9tgR7kffv0tI8etf6ULyzUXzjxhPGlrFU5ab5Xduyw/tkmRJatW/273sjNDb4wEMc4FffkaAp5/Lhy8cgMzJxpG60pliMd+4NLe/9+YO5c4IUX9LBzzjHHOXJENXeqV081vwOA886zTu/ZZ5WnwgYNgGoWzjOJgH37gMOHzV7+NP7yF9+0L7tMzVNSgDp1zNvq1/e1NwRW7/0Ldh22OSZANcHLzgaaNvXdptlnx4UX6sd03nlq3ZvUVKBWLcf2WhEJB5W2pKSoawfoTWRr11bhQnS57DLgggvst1eurKYQOXhQtUpNNJJD3F9+GWjdGlizBhg3zjLK4cefwSmUw0Gkm8IX4Xr/aaenK3F8/XXdBWzr1uY4J0/qy2+/re4GO3EvVcpa1I1ceKFq75yW5j9eDGnUKIC2EgHVq8fKnKCZNAm49FLg++8jm+6X6KoWSpXS20k3bhzZTARXueIKNSUaySHums/qQ/ZjipQqW+bMcnkcRxUcQAXk4ybMPxO+rP8E9RUAoOghi84j/fqpUvqNNwIATlbzlEhvv12Pk5ISV6IsKJYtU/O1ayObbjd8gQrI96x0U/dHExlpMpk4fNhtC0IjOcS9jEe4P//cNkoK6b1JT6I8clEFJ1ABRSiNaegMAOCUVKB8eZTBKdx5YKRPGszAidKV8Ode9fk9d88VGPO/k2owBS/y8lTtS7QpKVGdJ5mjn1csOHkyutUnWs1JpChBCk6ggh5g1xNWEGJMcoh72bJqPnq0bRSjuHtzCmr/33eUYOpUoBBl8M23vqfmzTeBChWAzVvVNgJj865yAJQoDRqk986uX18NDBQU99wT5A7A//4HdOigBprx5tSp2PSYjyRduqjqE0EQwiM5xL1MGfttCxeq7W3a2EYp8ZyGGV8Xn9FX73+QR48CL76oljefo/xyfIK/YvBgJexvv61Ga/v4YxUnOxvBwayGMQuSnTvt8ytXDrj55qCTdJU5c9Q80l8iyfJlIwhOSX5xb9ECOHECKe1vNQVrPn8AYCQeBwAsQoszYd7ifscd+nLeuReDwPja8zPt00+BY8fUtoICZybn5ACLFjmLq/Hjj2qYSiNaNYOdeC1cGFweblFYaG4cFC0xjnS1jCDEK6luGxARrMT90kuBzqouHSkpKO31GjOKxzy0BsGsJqW84hvH9vWu6jh0SG/95r2fHZmZqk4+GBG71fN+Mo67HEjcE4XHH1cvSQ0puQtCeCRHyV2rczeybZupmOtdYjv/fP9JaiI9frwqtRs9nXqXzlNTdXF32qw5Uj9boyHu+fmRbzIYiIkTzevex7NuHdCjB1BcjJDQ0pOSe3SpU8fs4TYQa9f6bQchhEHii/uBA8BI35YtgRgxwv/2/fuBv/1NufqeNcu8bcsW83qZMrq4BxKPkydV65ZIEQ1xf/hhoF07vT4/2ixY4Osm3Pt4unVTLs43bQovr0hzb+/7AAAa90lEQVSKu3wN+PL778H9b2rYEHjwwejZczaT+OLet6/jqN98oy9Xrhy49G73AjD2WQLU2ANOq2X691etWzTCHe9Dyy+SQqONX+J9nP5YsiR0GzzdBkx4n5d4LHmLuAvxTOKL+6RJvmE233kdOwKrV+tVDqE2EywsNK/n5zuvlvEu9XunBQB796pmjE7w7vEeCZx+hWhMnAhcd51v1Uo4eAuntu70n0ag9CKBMU0R+vhl/frQB1xLZBJf3K3wowANG+o/Ji++OLTkv/3WvD5wILBnj1rOyQHGjLHf11swtRdMRoZyO8Os7LLoF3UGrbmlMT1m9VJ48cXwhT5Ycdd6f+7dG16+RrzFMlib7IhWtUy8i/sDD1iXg84G6tdX09lGcol7166qF4zdmJpeeNelh8Mff6j5oEHBDXupifuaNaqtvCZiX31lv0+vXvqy9hXCDDzyCPDGG2o40HDQbHD6ZaM1A/XunLlihRLTUOru7aplQi25RwOjjfEu7hMmAPfd57YV0eH774HffnPbivjD0aNCRO2JaDMRbSOigRbbHyaiA0S0yjM5HKU3wnz5pVLFChUCx0XYTgZNrFplHZ6TY163K7lrOG0N0q+fmms3NbP+U9L752SwaKJlVWVkhZ3wfvKJmhv/dThFS7OoCNi8OfySe7SrZfx9Lf35p2q8FQ/8/rvbFkSedu2AZs3ctiL+CCjuRJQC4L8AOgCoB+A+IqpnEXUyMzfyTJ9E2M6okBqDVv7ePUS9xclbQDVxJ1I+zH74wTrdIUPM68z68WglaY233lLbvcPtMAprMPG90f4/9O4N5OY6S8s7zWefBerW1Uv/4ZbcnbwcvvsO+OmnwPGcVstUqwZcfnng9GLBTTe5bYEQK5w8Ks0AbGPmHcx8GsAkwONpK8EhirKPb/g23bMquf/4o3kdUGJRpw7Qtq2zao2SEuUGHlCu1Y0u1J97Tonieefp1Uf+0Gx0WnL33k/DKMRvvx1cWlpJWDs3kfxhHIgOHcw9mO1IpDp3DT+OU+OWI0ei/5wmI07EvRoAY8vV3Z4wb+4iojVENJWIgnWZFT61a4e8m3cpOJJoD/2GDco/jZW432rwjGCsltnvGVfEiUsDZt016cGDwPbt1vGCeUicirvHS7IP4ZSytfPmbUOoIhqLapm8PPVDO9iXYjS4+WbghhvctiIyZGY6cyY3Y4Z6EQiKSP2emgmgFjM3BPADgE+tIhFRTyLKIqKsAwcORChrD2EoyZNPRtAOG66+2tqJl7cQfPSRbxx/rnM0mCPf69WpSGn+zl580VwyDGcQIjtxD7cEH83WMs8/r35oDxqk7B42zLc66sQJ9aJfuhQYMCB6Jf758639CuXnA0OH+t83OxsoXz7yvu9Dxa6gAgDTpunLnTuH4Ik1iXGiiHsAGE9ZdU/YGZg5l5m1ltmfALAYaw1g5hHMnMnMmenp6VZRnDN8uPlJtRqazSHRHglNe4BXrvQVF+8RXl56yXf/Dz7wdRhml4dTWwBV0tEGD7Ii2BLon38qgdMI9L49dQr4+WfrbXY/deOp+sO75H7ihFp+/XXlhPTJJ4ErrzTvc9996kXfvDnw/vvqHLz2mv3XTzQI1O9v+nTVgc3Y5DZUtm5V9/yMGeGnpbF/v/rJfvSoahxnJD8/cvkkOk7E/TcAlxNRbSIqA6A7ANOlIqKqhtVOADZGzkQbnnjCvP7ll1HPMlSMAhWKz5ahQ4F//MN/HE1YgqFzZ+Vi17s0bCy5b99u7hP2/vt6/AED9JEHNTRBHzfOLFiHD6t0J0xQ1Ux9+ig3B3Y/+JyU3Fu2BKpWhSOMPVyZ1Us03JYj/urcf/lFzb1L7nPnmtdHjQJefVUJvJusXq2GEygqCtwxrrDQ+YtfGyRtwgQ9jDl0H0EAULOm+skeyIb9+9Vxna0EbC/CzEVE1BvAHAApAEYz83oiGgQ1CvcMAH2JqBOAIgCHADwcRZt9uf564KKLwkqiUiVVEogGsShNeIusHa1bq/dg1656WHGxdSm7sFANB2psZTNggOqN2rixEnpvKlRQXygPPWQO1wYYfv99JfpW1U9GnNS5e7tMnjhRVX1Z3QrafuvWqVLf66+rJporVvi3w4mN2nKgr4pWrXy/3LR7w/giPHJEVYs4qY6LFPffr6qLXnlFvxfsxD09XT0rq1apznd27Nql/kMA+ssOUEMev/56cO4tjGj/oAKd74YNfZsin004qqhm5tnMfAUzX8rMb3jCXvYIO5j5OWa+mpkzmPlmZg7TvVOQRKB3xpEjZp/tTnBaaoy3IVXvucfsO33dOrOAawJ0113WzSfLlgVmzrROOzXVup29VlJLTQUWLw5so/bgeqdVUqK+ZNatM4cfPqwEyui3xyq94cP1F0a4/QH++1+zXYH45ZfA/RwA4IILfKsbjDz1lPNGAN69qY3k5OguI7Trk5IS2F+RVghq1Mh/3rVqKed7gKqy0xg2TM2N91Yo1W2BSu5ns7ADydJDNUJFnGB/tsXKa2KkKSlRpS+NJk3sRdFuf61E5k1qqrUHZk3EUlL8u2cw5gFYl9z79gUaNLBO38lPZc0ffij/4F99Va9aM/5fSEtzlrd3dYSx6auR2bOt958/Xw2tqHViC8S2bfbCedtt6oWYm2v2jRSo5B4uVi+PUOr3E20IyViTHOJeunREkgkk7t4PlNtd4Z1WxThh0SJV1VKxYuDzUFxs/xN65kxr4ddK605/Xo8YoTxoejcDtRMcTSgOHlT/Ed56yz5trWOY1fXzV03DrOrG27XThwM02mXsr2CH949TTew1+wOVYFeuNK+fPOn/C+TYMeCFF6y3aX0eiorMJXft+n/yiaqqiTRWdfpPPKHWT51SLoBr1VJVPsYWzsePA//+t77ur+Qey34RgHqxEznr/BYzmNmVqWnTphwWejUn8+jR4aXl4dFHzckapxdf9M22pMQ+fjJPixczP/lkaPveeGN4ec+b5xtWvjzzyJG+4RrHjtmn16aN/W3lzejR+rZevSJzLp9/Xs3/9jeVx/bt9vkzM7/2mnk7wFy3rjnOK68Ezrew0Dp82zbm7t319QceYF65kvnOO5lPn7Y/R2vWMB8/7mubcTp4UIVfeKFa37PHvP30aeavv7a3uV498/qtt1rHY2Y+fNj+fggW4/7r1jE/+KA6ViNTpqg4Xbtap1FczDxxopqHC9S/TgSaAkaI1hRRcf/kk/DS8pCXxzxggLrp2rc3ZzF9ujnbq67yNeNsmRYudC/vVq2cx921S12jESP8x7O7rby57bbIH88//6kv169v3paVxfzttyrvdetUYeKll8z2acvLl6v1o0ed5fvJJ9bhCxaY1/v21ZdXr7Y+R5qo3367tW3a9NZbKvyii9T6H3+Yt584wfzVV+GfU7vz8MQTzEeO6NezoIC5dGnmceP0sJwc5j599BeZ8TiMy506me+NadNU+B13qPVly/Rrx8w8dKjaPnKk730VLE7FPTmqZSL0DVaxompPfvHFqkVJu3aqKdW4cb4/W2P92RdPhNOMLVyCcSusdU5zOmh5IM47LzLpGDHWG3v/JM7MVP9CZs1SLmvHj7e/75o2Vfest2dOO+z+mbRqZV43jj+QkqK8l3qjHUMgL6taNZg2976Piouj+1wNG6YG6Dl1Sp2r779XVTv//Kcep18/9cM+ULt87+2aX6eZM1UVTbNm5v9YffqoudaLPBaIuNtQvrxyINWwoRpqT6sn1Oog3RQ4t7EaOSlWaE0qnaDVyTod+MSbVavUde/USd1i0ejs5uQ++t//1PyJJ8yjg3n/G7n7buf5Ov1fYxwScsIE36aPfftan1+roSQ1EdXE3bspZFGR/0c5mAYP/tJp21adKyvP4Fp/EatrPX++fZrG337GZsBDhwIbDb1+ypZVPqBiUTgUcQ8S7SL6y7JOndjYIvjn++/Vj9xQfb00bqzmM2cCVapEzi4jTsRda86Yn6+GDI4EThzIefPmm75hQ4daDwLir4OY1qqoqVc/9kDizhzYRkB1FDO6JfBmwQLfsGefVV8e2leIVRsNK/chGsYGe8Yf3H37Atdco6//+adqQv3OO/ZpRQoR9yCxayZ29926J8FwPSv4w6nf6gYNfLu+n41Mnx64VdO4caoHsL920YcPR+c2C9SZKxGwEnc7f01LlujL3iX3tWsjU4XWpo0auMYp+/crr6V33KEXBIJ1B258GXi3XjK2kNK+/L1bW0WDGHg0jwGBelNEEO0iavNSpdRDP2WKKmH9+KPqGFKnjir5eTddi1T+gWjZ0v9n5NkCc2Bx1z6jA/l3OZur4vwRzMhf111nv6116/BtCQXjddXEXXvOAvnhGT5cvZCMpXN/o0Jpnf+i7c8KSJaSe4sWMcuqenXVfVobXWjLFuDrr9Vyhw5KTGrXVnNjm9xIEai/ltYuOCXl7P7pq+FE3DWGD/e/XcQ9MrRs6bYF9mheTTVx9+dBc/Jk9R/k738P3tVwLPrIJKa4O+m/HiWIVEcWbWSdSy8F7rzTOu4554SXl5WQBxIYrV4yNRWo5xkvK5ifkMlGSUnkHqQ49k2XUFi5Io4XtNZATn7edu+uL99+e3D5SMndivx85SgsAdDe/jVrBvahbaRlS2DqVGuPdlY/g4wYu5F/+qn6qWgclelsgzlyPtyd/tATEp9of/WKuFsRS8fXYaJdwCpV1DiiGuPHqx9pdoNqf/21ctpVty5w1VXWcexqorT4556r2u0bR3kKhlAGtY5H9uzx9R4pCIFo29a6TX+kiIW4J94P1VAcl7uE9sfd28FRmzb+xxYxNrv79VflJVBjwgTl7MmqbfGCBap52UsvAb16hW43oAaU6NDB16tgRoby+zF9ulofOlS1JHn55fDyixaTJ7ttgZCInDrl351xuEjJ3YoEKrlfeaXqEffGG+Zwf3XA3j+bjHEPHNDr0a2ajN1wg+p89Z//2Lu3N/7Vr1HD3o5zzvEtvW/bpr42NKdcNWqoL5J77/Xdv2FD+7QFe8IYUCxi1K3rtgXJj4i7Fd4jXwwY4I4dDqhQQZVqb7vNHG53Yfv3t69TL1NGleivvFJVvYTiETItDVi2TF/X/gNUrOgbt2xZVVdt/EIoV07fBuh12TVq+I5dGWr9dNOmQI8eoe0brwTT7yEexgBdsEDvwJVohNuIIVbUrBn9PBJP3L1L7oEGF40jtDa+Vv7O331XlfC9f/6VL69K4aNGqfVy5VRHiDZtgHnzgLFjVSuOQGJ/8KBvr8E77lA+Mqza5WoPibHFjvYVodmvCX+5cqrHo7HHofdxHDnibIjB2bOTS9yZzdVqgOqa/9RT9vHdJj09uLbr8UQsR68Kh0DDZkYEJ97FojGF7BVy+nTdNduIEaGl4RKbNimXpkbsPBBGE6s8i4pUWMWKumtW7/j796v1wkLmSy9VHvyMNGqkxx040NdTH7PaZ+tWX499DRsqT3rMvp4Jtek//9GXmzYN33tgLCZmda6MYYsWqfAJE3zjN2kSXn7duql5x47B7bdpk++1+uAD5tmzI3MePvrIOjwjI3LnunJl5tq13b/mTu+L0J/fZHX5O2OGfob++CO0NOKIK64I/2IHi90NNmSIesi9+eMP5mHDAqe7eLGe9rBh/m9mLbxaNTWfMUPfVlKi3MN6PxA5Ocw1aqjl339Xbl3HjWNOSQn+4Ro82Dfsu++c71+2rL7cuzdzy5b2D/G115rDDh/Wj3PzZubdu/VtxhdkIGG0mkpKmLOzmefOdb5P7dr+7wsn/uG9r5NxXXvUJ09m/vVX87ZFi4K/dnbT3LnMl1wSufSiNa1ZE/hZ8kfyijuzfpa0pySByc1lXr8+tnnu3Kl81keD3FzlR/30aSWgx44pv9neXHeduoSnTqmHvqTEN86yZaqUX7GiipuXx/z442rZ+HVhHMBCm+wGowCYy5Rh/uIL63169zaLbe/e5jg338w8fz5zhw56GLM6jr17fdNkVi8gQKVrhXHQl1dfVfPzz9fDghkURqOgwHfbnDlqnp2th02ZopeRvNPQ+OEH37Q6dVJz7Tp622B82Q0caE5PC//0U9+w5s39H9/YsdbhQ4aodKpWdX6u3Jgiwdkh7oWFoachuEpBgSp5OyEri/nvf1cid+qUGi3IyJ9/KpG56y51W7z+ugq3e8Cys5nXrlXLxoE8jBjD7ryTzwivRn6+//1SU4N7mPv2VdVRxcXMBw4wP/ecOX1t+emnfb86pk61tqW4WInhkiXqJWkMB9QXj5F169RIV1bk5TGXKqX2q1xZVeNt3qyuSZcuvgL25Zf6+vPPm9P64QfffNLT9X3XrmUeNUrfX3uBaCNOzZxpfvkBqgqJmblKleAF9/ff9eWePe3jDRqkzsNjj5nDJ0xgvv565vvuU+s1aqiX5p9/ml82kycHuAkccnaIuyD4weoBLV9e337ggBKn9et9P5UBNSQgs6pmeO45JWhGDh9WXypGjh9XJfS8vPA+LIuKmFetUkJmPBZmZfOQIXpYSQlzv35KxJ0yZIgS0WDQqlDq1PFNC2AeM0YJN7P6OtTs04ao9Me+fWo4PyPGvH79VV0vjcaN1XatymjzZhWuif7SpeZ/OwsXMr/wghoqUavaGz1ajX7FrL7IAOaTJ1U8q3tHG7GpqEh9OS5frgobGlZfn/n55tGfIkFExR1AewCbAWwDMNBie1kAkz3blwKoFShNEXch2kyfbj0urhNKSqwfVrf45htdODWefda3VBxNTp9m7tGDecsWc3hJiRJzb9avV+c72JeIxv79qlrPCq1abMMGc/jkycw1a+of9dnZvnFOn9YbB2gUFurVh5MnW4v72LGhHUekcSrupOLaQ0QpALYAuBXAbgC/AbiPmTcY4jwJoCEz9yKi7gC6MLNF1xadzMxMzsrKCrJtz5kM1TyA7YLw22/KB/7UqcrnfsWK9kPMCYlDbq5yn/vww9FJf8MG4LHHzP7np0wJbrSraEFEy5k5M1A8J+4HmgHYxsw7PAlPAtAZwAZDnM4AXvUsTwXwERERB3pzCEKUueYavQwwerR/f+JC4pCWFj1hB1RP8EcfVeJer54a/KZr1+jlFw2ciHs1ANmG9d0ArrWLw8xFRHQUQBqAg5Ew0odZsyI36rFw1hDM6DyC8PjjakpUYuo4jIh6AugJADX8OTYJhHd/fkEQBMGEE/cDewAYPV5U94RZxiGiVACVAOR6J8TMI5g5k5kz06M50KggCMJZjhNx/w3A5URUm4jKAOgOYIZXnBkAeniW7wYwV+rbBUEQ3CNgtYynDr03gDkAUgCMZub1RDQIqknODACjAIwjom0ADkG9AARBEASXcFTnzsyzAcz2CnvZsFwA4J7ImiYIgiCESuK5/BUEQRACIuIuCIKQhIi4C4IgJCEi7oIgCElIQN8yUcuY6ACAXSHuXgXR6v0aHvFqFxC/toldwSF2BUcy2lWTmQN2FHJN3MOBiLKcOM6JNfFqFxC/toldwSF2BcfZbJdUywiCICQhIu6CIAhJSKKK+wi3DbAhXu0C4tc2sSs4xK7gOGvtSsg6d0EQBME/iVpyFwRBEPyQcOJORO2JaDMRbSOigTHO+xIimkdEG4hoPRH184S/SkR7iGiVZ+po2Oc5j62biahdFG3bSURrPflnecIqE9EPRLTVM7/AE05ENMRj1xoiahIlm640nJNVRJRHRP3dOF9ENJqI9hPROkNY0OeHiHp44m8loh5WeUXArneIaJMn76+J6HxPeC0iOmk4b8MN+zT1XP9tHtspCnYFfd0i/bza2DXZYNNOIlrlCY/l+bLTBvfuMScDrcbLBOWVcjuAOgDKAFgNoF4M868KoIlnuSLU2LL1oIYY/IdF/HoeG8sCqO2xPSVKtu0EUMUr7G14BjQHMBDAYM9yRwDfAiAAzQEsjdG12wegphvnC0ArAE0ArAv1/ACoDGCHZ36BZ/mCKNjVFkCqZ3mwwa5axnhe6Szz2Eoe2ztEwa6grls0nlcru7y2/wfAyy6cLzttcO0eS7SS+5nxXJn5NABtPNeYwMx7mXmFZ/kYgI1QQwza0RnAJGY+xcy/A9gGdQyxojOATz3LnwK40xD+GSuWADifiKpG2ZY2ALYzs7+Oa1E7X8y8AModtXd+wZyfdgB+YOZDzHwYwA8A2kfaLmb+npmLPKtLoAbIscVj23nMvISVQnxmOJaI2eUHu+sW8efVn12e0nc3ABP9pRGl82WnDa7dY4km7lbjufoT16hBRLUANAaw1BPU2/N5NVr79EJs7WUA3xPRclLDGQLAhcy817O8D8CFLtil0R3mh87t8wUEf37cOG+PQpXwNGoT0Uoi+pmIbvCEVfPYEgu7grlusT5fNwDIYeathrCYny8vbXDtHks0cY8LiOhcAF8C6M/MeQCGAbgUQCMAe6E+DWNNS2ZuAqADgKeIqJVxo6eE4krTKFIjeHUCMMUTFA/ny4Sb58cOInoBQBGA8Z6gvQBqMHNjAAMATCCi82JoUtxdNy/ug7kAEfPzZaENZ4j1PZZo4u5kPNeoQkSloS7eeGb+CgCYOYeZi5m5BMBI6FUJMbOXmfd45vsBfO2xIUerbvHM98faLg8dAKxg5hyPja6fLw/Bnp+Y2UdEDwO4HcADHlGAp9oj17O8HKo++wqPDcaqm6jYFcJ1i+X5SgXQFcBkg70xPV9W2gAX77FEE3cn47lGDU+d3igAG5n5PUO4sb66CwDtT/4MAN2JqCwR1QZwOdSPnEjbVYGIKmrLUD/k1sE8tm0PANMNdj3k+WPfHMBRw6djNDCVqNw+XwaCPT9zALQlogs8VRJtPWERhYjaA/gngE7MfMIQnk5EKZ7lOlDnZ4fHtjwiau65Rx8yHEsk7Qr2usXyeb0FwCZmPlPdEsvzZacNcPMeC+cPsRsT1F/mLVBv4RdinHdLqM+qNQBWeaaOAMYBWOsJnwGgqmGfFzy2bkaYf+T92FUHqiXCagDrtfMCIA3ATwC2AvgRQGVPOAH4r8eutQAyo3jOKgDIBVDJEBbz8wX1ctkLoBCqHvOxUM4PVB34Ns/0SJTs2gZV76rdY8M9ce/yXN9VAFYAuMOQTiaU2G4H8BE8HRQjbFfQ1y3Sz6uVXZ7wsQB6ecWN5fmy0wbX7jHpoSoIgpCEJFq1jCAIguAAEXdBEIQkRMRdEAQhCRFxFwRBSEJE3AVBEJIQEXdBEIQkRMRdEAQhCRFxFwRBSEL+H6A1kO4IPbwjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe1cd360f90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##################################################################################################\n",
    "def AddLeNetModel(model, data):\n",
    "    # Image size: 28 x 28 -> 24 x 24\n",
    "    conv1 = brew.conv(model, data, 'conv1', dim_in=1, dim_out=20, kernel=5, \\\n",
    "                weight_init=('XavierFill', {}),\n",
    "                bias_init=('XavierFill', {}))\n",
    "    # Image size: 24 x 24 -> 12 x 12\n",
    "    pool1 = model.net.MaxPool(conv1, 'pool1', kernel=2, stride=2)\n",
    "    # Image size: 12 x 12 -> 8 x 8\n",
    "    conv2 = brew.conv(model, pool1, 'conv2', dim_in=20, dim_out=50, kernel=5, \\\n",
    "                weight_init=('XavierFill', {}),\n",
    "                bias_init=('XavierFill', {}))\n",
    "    # Image size: 8 x 8 -> 4 x 4\n",
    "    pool2 = model.net.MaxPool(conv2, 'pool2', kernel=2, stride=2)\n",
    "    # 50 * 4 * 4 stands for dim_out from previous layer multiplied by the image size\n",
    "    fc3 = brew.fc(model, pool2, 'fc3', dim_in=50 * 4 * 4, dim_out=500, \\\n",
    "                weight_init=('XavierFill', {}),\n",
    "                bias_init=('XavierFill', {}))\n",
    "    fc3 = model.net.Relu(fc3, 'relu3')\n",
    "    pred = brew.fc(model, fc3, 'pred', 500, 10)\n",
    "    softmax = model.net.Softmax(pred, 'softmax')\n",
    "    return softmax\n",
    "###################################################################################################\n",
    "\n",
    "\n",
    "#TRAINING and TESTING\n",
    "train_model = model_helper.ModelHelper(name=\"mnist_train\", arg_scope=arg_scope)\n",
    "data, label = AddInput(\n",
    "    train_model, batch_size=64,\n",
    "    db=os.path.join(data_folder, 'mnist-train-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "\n",
    "softmax = AddModel(train_model, data)\n",
    "AddTrainingOperators(train_model, softmax, label)\n",
    "\n",
    "workspace.ResetWorkspace()\n",
    "workspace.RunNetOnce(train_model.param_init_net)\n",
    "workspace.CreateNet(train_model.net, overwrite=True)\n",
    "total_iters_x = 2000\n",
    "accuracy_x = np.zeros(total_iters_x)\n",
    "loss_x = np.zeros(total_iters_x)\n",
    "printmd('**Training status: Running**  Wait till the process is completed')\n",
    "for i in range(total_iters_x):\n",
    "    workspace.RunNet(train_model.net)\n",
    "    if(i%100 == 0):\n",
    "        print \"    Completed iterations:\", i, \", Total iterations to be completed:\", total_iters_x, \\\n",
    "        \", Current Loss = \", workspace.blobs['loss']\n",
    "    accuracy_x[i] = workspace.blobs['accuracy']\n",
    "    loss_x[i] = workspace.blobs['loss']\n",
    "#print \"Training completed\"\n",
    "printmd('**Training status: Completed**')\n",
    "\n",
    "#print \"Testing status: Running\"\n",
    "printmd('**Testing status: Running**  Wait till the process is completed')\n",
    "test_model = model_helper.ModelHelper(\n",
    "    name=\"mnist_test\", arg_scope=arg_scope, init_params=False)\n",
    "data, label = AddInput(\n",
    "    test_model, batch_size=100,\n",
    "    db=os.path.join(data_folder, 'mnist-test-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "softmax = AddModel(test_model, data)\n",
    "AddAccuracy(test_model, softmax, label)\n",
    "workspace.RunNetOnce(test_model.param_init_net)\n",
    "workspace.CreateNet(test_model.net, overwrite=True)\n",
    "test_iters_x = 1000\n",
    "test_accuracy_x = np.zeros(test_iters_x)\n",
    "for i in range(test_iters_x):\n",
    "    if(i%100 == 0):\n",
    "        print \"    Completed test set:\", i, \", Total sets to be tested:\", test_iters_x\n",
    "    workspace.RunNet(test_model.net.Proto().name)\n",
    "    test_accuracy_x[i] = workspace.FetchBlob('accuracy')\n",
    "#print \"Testing completed\"\n",
    "printmd('**Testing status: Completed**')\n",
    "# After the execution is done, let's plot the values.\n",
    "pyplot.plot(loss_x, 'b')\n",
    "pyplot.plot(accuracy_x, 'r')\n",
    "pyplot.legend(('Loss', 'Accuracy'), loc='upper right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Effects of Initializers**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" cellpadding=\"3\" cellspacing=\"0\"  style=\"border:black; border-collapse:collapse;\"><tr><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Initializer&nbsp;Type</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Num&nbsp;iterations</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;LR</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;Loss</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Loss</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;Training-Acc(%)</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Training-Acc(%)</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Testing-Acc(%)</b></td></tr><tr><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">Constant&nbsp;-&nbsp;high</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">43.1735</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">41.7344</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">6.2500</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">9.3750</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.3200</td></tr><tr><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">Constant&nbsp;-&nbsp;zero</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2.3026</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2.3023</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.9375</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.9375</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">11.3500</td></tr><tr><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">Xavier</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">3.2040</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.1746</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">12.5000</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">92.1875</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">94.2800</td></tr></table>"
      ],
      "text/plain": [
       "<ipy_table.ipy_table.IpyTable at 0x7fe1cd360390>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "printmd('**Effects of Initializers**')\n",
    "\n",
    "initializers = [\n",
    "    ['Initializer Type', 'Num iterations', 'Init LR', 'Init Loss', 'Final Loss', 'Init Training-Acc(%)', \\\n",
    "     'Final Training-Acc(%)', 'Final Testing-Acc(%)'],\n",
    "    ['Constant - high', total_iters_c_h, base_lr, loss_c_h[0], loss_c_h[total_iters_c_h-1], accuracy_c_h[0]*100, \\\n",
    "     accuracy_c_h[total_iters_c_h-1]*100, np.mean(test_accuracy_c_h)*100],\n",
    "    ['Constant - zero', total_iters_c_m, base_lr, loss_c_m[0], loss_c_m[total_iters_c_m-1], accuracy_c_m[0]*100, \\\n",
    "     accuracy_c_m[total_iters_c_m-1]*100, np.mean(test_accuracy_c_m)*100],\n",
    "    ['Xavier', total_iters_x, base_lr, loss_x[0], loss_x[total_iters_x-1], accuracy_x[0]*100, \\\n",
    "     accuracy_x[total_iters_x-1]*100, np.mean(test_accuracy_x)*100],\n",
    "\n",
    "];\n",
    "make_table(initializers)\n",
    "apply_theme('basic')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. MSRA+Gaussian initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Training status: Running**  Wait till the process is completed"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Completed iterations: 0 , Total iterations to be completed: 2000 , Current Loss =  3.0503173\n",
      "    Completed iterations: 100 , Total iterations to be completed: 2000 , Current Loss =  1.2514492\n",
      "    Completed iterations: 200 , Total iterations to be completed: 2000 , Current Loss =  0.52981883\n",
      "    Completed iterations: 300 , Total iterations to be completed: 2000 , Current Loss =  0.41093937\n",
      "    Completed iterations: 400 , Total iterations to be completed: 2000 , Current Loss =  0.30870074\n",
      "    Completed iterations: 500 , Total iterations to be completed: 2000 , Current Loss =  0.37612173\n",
      "    Completed iterations: 600 , Total iterations to be completed: 2000 , Current Loss =  0.21828662\n",
      "    Completed iterations: 700 , Total iterations to be completed: 2000 , Current Loss =  0.3924354\n",
      "    Completed iterations: 800 , Total iterations to be completed: 2000 , Current Loss =  0.39088798\n",
      "    Completed iterations: 900 , Total iterations to be completed: 2000 , Current Loss =  0.34615397\n",
      "    Completed iterations: 1000 , Total iterations to be completed: 2000 , Current Loss =  0.22569561\n",
      "    Completed iterations: 1100 , Total iterations to be completed: 2000 , Current Loss =  0.15061213\n",
      "    Completed iterations: 1200 , Total iterations to be completed: 2000 , Current Loss =  0.2098943\n",
      "    Completed iterations: 1300 , Total iterations to be completed: 2000 , Current Loss =  0.17711574\n",
      "    Completed iterations: 1400 , Total iterations to be completed: 2000 , Current Loss =  0.1406072\n",
      "    Completed iterations: 1500 , Total iterations to be completed: 2000 , Current Loss =  0.33077425\n",
      "    Completed iterations: 1600 , Total iterations to be completed: 2000 , Current Loss =  0.35632062\n",
      "    Completed iterations: 1700 , Total iterations to be completed: 2000 , Current Loss =  0.12004657\n",
      "    Completed iterations: 1800 , Total iterations to be completed: 2000 , Current Loss =  0.135901\n",
      "    Completed iterations: 1900 , Total iterations to be completed: 2000 , Current Loss =  0.1872685\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Training status: Completed**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Testing status: Running**  Wait till the process is completed"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Completed test set: 0 , Total sets to be tested: 1000\n",
      "    Completed test set: 100 , Total sets to be tested: 1000\n",
      "    Completed test set: 200 , Total sets to be tested: 1000\n",
      "    Completed test set: 300 , Total sets to be tested: 1000\n",
      "    Completed test set: 400 , Total sets to be tested: 1000\n",
      "    Completed test set: 500 , Total sets to be tested: 1000\n",
      "    Completed test set: 600 , Total sets to be tested: 1000\n",
      "    Completed test set: 700 , Total sets to be tested: 1000\n",
      "    Completed test set: 800 , Total sets to be tested: 1000\n",
      "    Completed test set: 900 , Total sets to be tested: 1000\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Testing status: Completed**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7ee1cc605850>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJztnXmY1FTWxt8DNCKCIIvKpoCggkqztKKAjIIiKsowKuKCgAsybjjO6ODoxyjO6OCuyIgwLOIwoKAgm7IIiChb07bsS7M3NFu3LG3T0N11vj9uhaRSSVVqTVX1+T3PfZLc3Nyc3CRvbu5KzAxBEAQhtajgtgGCIAhC9BFxFwRBSEFE3AVBEFIQEXdBEIQURMRdEAQhBRFxFwRBSEFE3AVBEFIQEXdBEIQURMRdEAQhBank1onr1KnDjRs3duv0giAIScmaNWuOMHPdYOFcE/fGjRsjMzPTrdMLgiAkJUS020k4KZYRBEFIQUTcBUEQUhARd0EQhBTEtTJ3QRBSl5KSEuTm5qK4uNhtU5KWKlWqoGHDhkhLSwvreBF3QRCiTm5uLqpXr47GjRuDiNw2J+lgZuTn5yM3NxdNmjQJKw4plhEEIeoUFxejdu3aIuxhQkSoXbt2RH8+Iu6CIMQEEfbIiDT9kk7c168H/u//gEOH3LZEEAQhcQkq7kRUhYhWEdEvRLSBiF61CHMWEX1ORDlEtJKIGsfCWADYvBn4xz9E3AVBCEy1atXcNsFVnOTcTwHowszpAFoD6E5E15rCPALgV2ZuBuA9AMOja6ZOJW8VcGlprM4gCIKQ/AQVd1YUejfTvI5NwXoC+NS7Pg1AV4pRgZuIuyAI4bJr1y506dIFrVq1QteuXbFnzx4AwNSpU3HllVciPT0dnTt3BgBs2LAB11xzDVq3bo1WrVph27ZtbpoeMo6aQhJRRQBrADQDMJKZV5qCNACwFwCYuZSIjgGoDeBIFG0FIOIuCMnGs88C2dnRjbN1a+D990M/7umnn0a/fv3Qr18/jBs3Ds888wxmzJiBYcOGYd68eWjQoAGOHj0KABg1ahQGDx6MBx54AKdPn0ZZWVl0LyLGOKpQZeYyZm4NoCGAa4joynBORkQDiSiTiDIPHz4cThQi7oIghM3y5ctx//33AwD69u2LZcuWAQA6duyI/v37Y8yYMWdE/LrrrsPrr7+O4cOHY/fu3Tj77LNdszscQurExMxHiWgxgO4A1ht27QPQCEAuEVUCUANAvsXxowGMBoCMjAxz0Y4zg0XcBSGpCCeHHW9GjRqFlStXYs6cOWjXrh3WrFmD+++/H+3bt8ecOXNw22234ZNPPkGXLl3cNtUxTlrL1CWimt71swHcDGCzKdhMAP2863cDWMTMYYl3METcBUEIlw4dOmDKlCkAgEmTJuH6668HAGzfvh3t27fHsGHDULduXezduxc7duxA06ZN8cwzz6Bnz55Yu3atm6aHjJOcez0An3rL3SsA+IKZZxPRMACZzDwTwFgAnxFRDoACAH1iZrCIuyAIDigqKkLDhg3PbD/33HMYMWIEBgwYgLfeegt169bF+PHjAQDPP/88tm3bBmZG165dkZ6ejuHDh+Ozzz5DWloaLrzwQvztb39z61LCgmKUwQ5KRkYGhzNZR2YmcPXVwKxZQI8eMTBMEISI2bRpE1q0aOG2GUmPVToS0Rpmzgh2bNL1UJWcuyAIQnBE3AVBEFIQEXdBEIQURMRdEAQhBRFxFwRBSEFE3AVBEFKQpBX3khJ37RAEIfGZMWMGiAibN5v7XaY+SSvuknMXBCEYkydPRqdOnTB58uSYnSNRBxQTcRcEISUpLCzEsmXLMHbs2DNDDgDA8OHDcdVVVyE9PR1DhgwBAOTk5OCmm25Ceno62rZti+3bt2PJkiXoYegp+dRTT2HChAkAgMaNG+Ovf/0r2rZti6lTp2LMmDG4+uqrkZ6ejrvuugtFRUUAgIMHD6JXr15IT09Heno6fvrpJwwdOhTvGwbceemll/DBBx9E/fpDGjgsEUhLU0sRd0FIElwa8/frr79G9+7dcemll6J27dpYs2YNDh06hK+//horV65E1apVUVBQAAB44IEHMGTIEPTq1QvFxcXweDzYu3dvwPhr166NrKwsAEB+fj4ee+wxAMDLL7+MsWPH4umnn8YzzzyD3/3ud5g+fTrKyspQWFiI+vXr4w9/+AOeffZZeDweTJkyBatWrYpCoviSdOJesaJairgLghCIyZMnY/DgwQCAPn36YPLkyWBmDBgwAFWrVgUA1KpVCydOnMC+ffvQq1cvAECVKlUcxX/vvfeeWV+/fj1efvllHD16FIWFhbjlllsAAIsWLcLEiRMBABUrVkSNGjVQo0YN1K5dGz///DMOHjyINm3aoHbt2lG7bo2kE/cK3oIkl4bEEQQhVFwY87egoACLFi3CunXrQEQoKysDEeGee+5xHEelSpXg8XjObBcXF/vsP+ecc86s9+/fHzNmzEB6ejomTJiAJUuWBIz70UcfxYQJE3DgwAE8/PDDjm0KhaQrc9cm7zOkuSAIgg/Tpk1D3759sXv3buzatQt79+5FkyZNUKNGDYwfP/5MmXhBQQGqV6+Ohg0bYsaMGQCAU6dOoaioCBdffDE2btyIU6dO4ejRo/juu+9sz3fixAnUq1cPJSUlmDRp0hn/rl274uOPPwagKl6PHTsGAOjVqxe+/fZbrF69+kwuP9oknbhrOfcNG9y1QxCExGXy5Mlnilk07rrrLuTl5eHOO+9ERkYGWrdujbfffhsA8Nlnn+HDDz9Eq1at0KFDBxw4cACNGjVC7969ceWVV6J3795o06aN7flee+01tG/fHh07dsTll19+xv+DDz7A4sWLcdVVV6Fdu3bYuHEjAKBy5cq48cYb0bt3b1TUypqjTNIN+evx6OXuUjQjCImJDPkbGI/Hc6alTfPmzW3Dlashf7ViGUEQhGRk48aNaNasGbp27RpQ2CMl6SpURdwFQUhmWrZsiR07dsT8PEmXcxcEITlwq8g3VYg0/UTcBUGIOlWqVEF+fr4IfJgwM/Lz8x23ubci6YplBEFIfBo2bIjc3FwcPnzYbVOSlipVqvhM8B0qIu6CIESdtLQ0NGnSxG0zyjVSLCMIgpCCiLgLgiCkICLugiAIKUhQcSeiRkS0mIg2EtEGIhpsEeYGIjpGRNleNzQ25gqCIAhOcFKhWgrgz8ycRUTVAawhogXMvNEU7gdm7mFxvCAIghBngubcmTmPmbO86ycAbALQINaGCYIgCOETUpk7ETUG0AbASovd1xHRL0T0DRFdYXP8QCLKJKJMaf8qCIIQOxyLOxFVA/AlgGeZ+bhpdxaAi5k5HcAIADOs4mDm0cycwcwZdevWDddmQRAEIQiOxJ2I0qCEfRIzf2Xez8zHmbnQuz4XQBoR1YmqpYIgCIJjnLSWIQBjAWxi5ndtwlzoDQciusYbb340DRUEQRCc46S1TEcAfQGsIyJtCvO/AbgIAJh5FIC7AfyRiEoBnATQh2XEIEEQBNcIKu7MvAxAwFHUmfkjAB9FyyhBEAQhMqSHqiAIQgqS1OL+lV/VriAIggAkubjv2uW2BYIgCIlJUot7WprbFgiCICQmSS3uzzwDNGsGrF/vtiWCIAiJRVKK+xtv6OvbtwN/+Yt7tgiCICQiSSnud9zhu/3DD4DH444tgiAIiUhSivvp077bRUXAv//tji2CIAiJSFKKewULq7dsib8dgiAIiUpSint6OvDEE75+paXu2CIIgpCIJKW4A8Dzz/tul5S4Y4cgCEIikrTiXsk0Ko6IuyAIgk7SivuFFwL16+vbIu6CIAg6SSvulSoB+/YBF12ktidPBv71L3dtEgRBSBSSVtw1zj1XX3/xRffsEARBSCSSXtyrVnXbAkEQhMQj6cX99tvdtkAQBCHxSHpxf/lloEMHfVvauwuCIKSAuFeoAHTtqm+npUlvVUEQhKQXdwAYNgy44QZ9OzPTNVMEQRASgpQQdwBo315fl6IZQRDKOykj7hUr6uuFhe7ZIQiCkAikjLgbR4p86in37BAEQUgEgoo7ETUiosVEtJGINhDRYIswREQfElEOEa0loraxMdceY84dAHbujLcFgiAIiYOTnHspgD8zc0sA1wJ4kohamsLcCqC51w0E8HFUrXRAp06+26++Gm8LBEEQEoeg4s7Mecyc5V0/AWATgAamYD0BTGTFCgA1iahe1K0NwE03AV99pW/v2xfPswuCICQWIZW5E1FjAG0ArDTtagBgr2E7F/4fgJjTqxfQ0vtPsXBhvM8uCIKQODgWdyKqBuBLAM8y8/FwTkZEA4kok4gyDx8+HE4UQbngAn1dhgEWBKG84kjciSgNStgnMfNXFkH2AWhk2G7o9fOBmUczcwYzZ9StWzcce4MyZYq+vmtXTE4hCIKQ8DhpLUMAxgLYxMzv2gSbCeAhb6uZawEcY+a8KNrpmPPP19ffeMMNCwRBENzHSc69I4C+ALoQUbbX3UZEg4hokDfMXAA7AOQAGAPgCZu44sr48W5bIAiC4A6VggVg5mUAKEgYBvBktIyKJswABbReEAQh9UiZHqpGunXT141l8IIgCOWFlBT3oUP19fvvd88OQRAEt0hJcU9L890uK3PHDkEQBLdISXGvXdt3+69/dccOQRAEt0hJcW/aFHjnHX3788/ds0UQBMENUlLciYDnngMuv1xtnz7trj2CIAjxJiXFXaOSt6HnqVPu2iEIghBvUlrctTHei4vdtUMQBCHelAtxP3UKWLfOXVsEQRDiSUqL++7d+vpK8yDFgiAIKUxKi3t+vtsWCIIguENKi7sRZrctEARBiB8pLe433eS2BYIgCO6Q0uI+Z46+vnmze3YIgiDEm5QW98qV9fV37aYZEQRBSEFSWtwBoFUrty0QBEGIPykv7tOmuW2BIAhC/El5ca9e3W0LBEEQ4k/Ki7ux3L15c/fsEARBiCflStxzctyzQxAEIZ6UK3EXBEEoL6S8uJun3BMEQSgPpLy4E/lul5a6Y4cgCEI8SXlxB4A2bfT1kyfds0MQBCFeBBV3IhpHRIeIaL3N/huI6BgRZXvd0OibGRnGopmiIvfsEARBiBeVHISZAOAjABMDhPmBmXtExaIYUMlwlSLugiCUB4Lm3Jl5KYCCONgSF0TcBUEoD0SrzP06IvqFiL4hoiuiFGfUqFVLX8/IAB5/HChImc+VIAiCP06KZYKRBeBiZi4kotsAzABg2ReUiAYCGAgAF110URRO7Yzx44G6ddV6cTEwejTg8QBjxsTNBEEQhLgScc6dmY8zc6F3fS6ANCKqYxN2NDNnMHNGXU1t40CdOsDChb5+JSVxO70gCELciVjciehCItWanIiu8caZcLOXnnWW77bH444dgiAI8cBJU8jJAJYDuIyIconoESIaRESDvEHuBrCeiH4B8CGAPsyJN2OpeRiCL78Ejh1zxxZBEIRYE7TMnZnvC7L/I6imkgmNWdyLioAHHgBmz3bHHkEQhFhSLnqoAtYDiK1bF387BEEQ4kG5FncZVEwQhFSl3Ii7uUIVEHEXBCF1KTfiLjl3QRDKEyLugiAIKUi5FvdK0eifKwiCkICUG3GXMndBEMoT5UbcK1b099PEvaRE2rsLgpBalBtxN0+3B+ji/uqrwB13APPnx9cmQRCEWFFuxN2KBQuAI0eAHTvU9uHD7tojCIIQLcq1uAPAK6+4bYEgCEL0KfftRWbN0itbrYpuBEEQkpFyL+579ujrIu6CIKQK5apYZv/+wPtF3AVBSBXKlbjXq+e2BYIgCPGhXIm7IAhCeaHciXtGhv0+KZYRBCFVKHfivnw5cPq021YIgiDElnIn7pUqqZ6pnTv775OcuyAIqUK5E3cNEXJBEFKZcivuVojgC4KQKpRbcRchFwQhlRFxD+InCIKQjJRbcW/d2m0LBEEQYkdQcSeicUR0iIjW2+wnIvqQiHKIaC0RtY2+mdFn+HCgUydfP8m5C4KQKjjJuU8A0D3A/lsBNPe6gQA+jtys2JOWBnTr5rYVgiAIsSGouDPzUgAFAYL0BDCRFSsA1CSipBjFpU4d323JuacYq1cDeXnAnDnAhg3AW29Zz8iSkwNs3qzWFywI3MttyxYVHgC2b9ePc4OtW5U9ofDrr8CyZbGxxwwzMHeuWn73HVBcHNrx2dnA3r1qfcUKNbOORlERsHhx9Gw18s03QGkpcPQo8NlnwOTJwJgx6jqSCWYO6gA0BrDeZt9sAJ0M298ByLAJOxBAJoDMiy66iN1m9mxmdceU+/LLGJ/wwAHm7OzI41m2jPm33yKPx8zKlcy//hpZHLm5zBs26NtHjjCvXh38uKVLmYuK9O3Fi5lXrWKeMkXFsWKF8i8sVNdvR0GBOt/69b43V3NVq6r7wMy8aBFzSYm+b/VqtXzmGd84T59WYZn1sLt36+tG5s5l/uQTZf/WrcGv20xRkUqL335TywMHmGfNYh43jnnzZuasLObDh31tKS1lXrjQP67Fi5lPnVJpsXcv86FD+jErVjDn5zu3a80a5nXrmJ9+mnn6dOYTJ9S1fv65uk+zZzP/8IMK+9VXzD/9xPzpp+pcgwer5f3329+7zZvVMUbM9+XKK3VbundXflOmMM+fz+zxqH3HjvnGs2ePeh7z85Wda9Ywjx6trr+gQNm0f78Ku3Mn89ix1s8NoK6LWd2X/HyVvv37M69dq+7B2LHMW7b4XsOuXcyTJjG/9JJ6N6IAgEx2otuOAkVJ3I2uXbt2UbnQSFi4MM7ifskl6kTag8ishGvtWvXAbN4c/AHYs0fFYfw47tvn/1CZ2bpVhfN4lFC88Qbztm3qwT94kPm771S8HTvqxxw+rF7oHTtU/OaXzwotMXfsUNuXXaYL4JYtzP/7n1o/dIh53jx1zRMmqDAPPaSE9K23rF+ugwfVdQPKNo9HicuGDcxLljC/9579i2l2c+aoZadOut8HH6jlWWcpu/LymGfMYL7tNuX/4ovWcTGrj+LEif77NMFjVg/c998ru5csYS4rY16wQIV54QUlYg8/rI5r29Y/rurV1fKyy3T7AeZbb9WvyeNRgjdtmvJr1UoP16yZb3wtWqiPxbFjyobcXHXdzCqttY/00qX+tjRubJ0WWhoCzNdeax1mxAglesx6WjRtqvZ9/rlKh59+sj62oMDa//nnVXr26qW233yT+b//1fefdVbg52HMmODPzGuvqWfYap/2bgPqQ/f888x//7t/uFmzmH/8Mfh7FPAVi5+4fwLgPsP2FgD1gsWZCOL+ww++6T51SqnKvUaTzEzmo0dVjkE7kTF33Ly5tVgY8Xh0YV27Vg83ZYrvA2wW+JMn1XmZ9TCakAZy2dkqF2O175df1DX9/LPKRRuvc9Mm37DGXyPjS/Htt/bn1nJ5wdz+/SpH5CRsrN3Klczt2tnvX7qU+Z139O0hQ9SSyD+slag7dd26Mb/9duTXw8x8wQVqfd682KXbxInMH38cvfj69o39vX7ttejEc/Bg2JIST3G/HcA3AAjAtQBWOYkzEcRd+9vT3KY+r6gVrQhAY8MGXyHzUlZm8igtVTkhDbuv/NKlKgeyd6/1fi1yTZhHj1b+f/mLEtVgL2ZWlrJlwADll5ur769WLbKHsmFDfb15c3UNxg9XvNwHH6jf9Hiftzy4WAq6OOU2bgxbt6Im7gAmA8gDUAIgF8AjAAYBGOTdTwBGAtgOYJ2TIhnmxBB3YyYYYJ6Ont6V6XqgwkLld801PseuXKm8Fy5kVdxx8CDz0KHKU8v5jhsX3o0/cEDPhXzxBfPVV+v7/v3vwMdqv+s33sh8/vlq3Xyh4sSJc9eNGxe2bkU15x4LlwjibqxfApi/xh1qZcYM5uPHVTm1sQyRWeWov/mG3/rnKQaYXx20X9/fpYtaBip2cMPNnOm+DW464z0UJ86Jmz7d9483Evfoo/5+l10Wtm45Ffdy20MVAOrWBebN07crwAMAePX3WcC55wKXXgoMHux70D33ALfeikfeaQEAGDqqvr5v0SK1/OqrWJodOnfe6bYF7lKtmv2+66+Pnx3RZr1lv0IhXN56S1+/9FKgQYPI47z9duBji64/fftGHncwnHwBYuESIefOrBqMaB/T2bgt8Bf42DGfbZV6CZDLEGfttm5VxWVakzwrt22bKlbTWgUFiq937+jbWKmS/b6xY/VWTGbXs6d6gN1O4/z86MbXokV87c/LU5Xzu3aphguav0aPHvbHFhaq+rgTJ9TSKqdfUuJ7n0aNUs1ojS3mQgSSc3dGxYr6+lk4FThwjRo+m7WQHwOLkoRLL3XbAl/mzvX3a94cOP98tdTo2FFfr1cPaNYMqF8fuPJKoEkT1enJjt/9LrANFcJ4nQKlY61aQJcuwGWXqe2FC4Hx49X67ber5euv6+Fzc4GGDZ2fu3Zt52Evv1wtR470t3HdOv/w8+erzkDGv9gdO4D9+4ELLlDbv/zie8xFFym/H36wtkFLh2Ds2QNs2xY83NlnAxdeqJ6Diy+27sU4daq+bnw2hgwBzjkHaNlS/Rm2bKly+nl56vyVKqlw2vLJJ9XyttvUdcajx6STL0AsXKLk3HOzD3MtHOHKKOajODf+OR+3nNae2uzsWvAYXZ06/k2NYuW6dVPLmjXtwxQUqNwTwNyhg+5vRGsL/v77+v7rrrN+KIxxv/GGvl5SolpSWdmwYoVvUzy7dtVaJbfmRo7U1197Tf1JDB+utrXWUkVF6r5obN2q5/zmz1dhn35abf/xj77x79nj+3tqdEZ7P/9cVbzffrt12H/+Uy1Xr/ZtL8+sOlyZw69dq/YZ63s06tdX28aOYID6M7a6B5p77jnf7VmzrMMxq+bHZn+tFcTAgWp57732997OT2vSOmeO9bOjcfSob3PHkhK970eEQCpUbSgqYi4uVutbt9oLRryc+YGNh7vkEt+emeYXw7h9883+Yd58kzknxz5+c2cZO3f11czLlwcOs2mT6lhz/fVq26q1kEZOjvrFtnpBtfbz772nd/Bp08b6GdGO1zrxmOMrLFS/8UYBOXqUuV8/ffvGG62vx/zxHDVKLTt31tvWlpWF1lRu/Xpd7E+ftk4bK1uKi/VOP9u3q3Djx6ttrZPS4MGq+a3RplOnVPNcYw/XnBzVBFeLW7Pnyy/19NDQPnB5eXp47fwau3cr8TXa++STvtvGIqGKFdWyWjV1fFGR2jZ24mJW1+DxKHs1HTCiHWOkQQPftDT2wHYBEXcrtBveoIG6wdOnOxMhJyIV7rEej29vw8xM63A33eTf6mPuXNV9GmB+8EGVEw10rh499K7tzOoF0rrpNm+ud8vXwvfurXLExpzf11/rIrRokfV5nIr7iBF+9Rhn3K5dzN98o9+7zp2Vv1UZtBkr/z/9Sfm9844usAMHWj8n5uNzc1XTKjPGMtoTJ/S/ghEj9IzDvffqPUsB1YHtiiv0be1jNWiQs2fYCTk5qgOZ1gvUeE2Aup9al/vTp9XHwXhN2dn6H8X8+aGd25x2U6ao7Xvu0f3uvlv5HT+ul7Fbcfq0qgvR4vzoI//7vnKl6uV7+LD6SzlyRD9+82b9nbc7h5nt233/IJjVPdu509nxcUDE3Yw5p/ruu84EyIl79VXnYWvWVDmc3FzfDk/afqtfXID5sceUqJof7hdeUOv/+pcugIB1scB771mnzaZN6kXT0HoN/ve//vaZWbtWiUjr1nqYBx9Uyw0bfHNOgCo22LpVjfPh8dhf76lTPqcpvf537CfueXm+AqZRp46/rVpHMG18kOxs65wbsxJ/TfyCodlSVKSuxziOTlaWEqiDB5nT0/nMR+DoUdUFfccOXbCeeMLZ+cLll1/UeW691Vl487U4xfycfP+92n79dd2vqEiJNrMSzmBDZxQU6EVRq1czn3OOc7G2sinJEXE3Y/yCR9sZy2XNbt063+IfbdAnM9p+82+15t56S4Uz5qKZ9d/6qVNVuSug2ukfPKiHq19f5YLtxMyMx6PGZjDW6GdmKmG145VX9POdPKnCM6uyXED/8EyY4HtccbH19ZaW+gTbWkf9lRyY9oPKXQUS38OH9TFSjNcU4Zgelth8jPz47Tffj7mGVib9wQfRt81Mdrb6uMQSKyH96SeL7twRcOhQ8A+CkT17lEsRRNzN2P3+R8GdfmSQ/X6NYLkHbb/2q9+okSqvzMtTuR/jy7F9+5kca1mph398fQl7yjxKYIwCpsXZsmXEyacN9GdLaamqEDRW/DGr61m82HdpPk77AC1ZwiWXXaGng4H1VdSYKzlTwshNxhKbj1FIaIOIpQI7dyZUEUYqIuJuxMmIbxZuzUjryr6s1gN8tjdc1J3f7pvND2EC57QwtTbQcCruzKocsaDA0aVpxfCTJvnvuxtfqJ39+4eQWMHNizrLl58ZTC3jkgJuj+V+QWadq+oWtnwbXouD3bvVmFwRjNdkSRZaW36MBCFWOBX38tHOffTokA85jupo9+S1lvvWN+7hG7ZCTeTWTsdE9MPcByc7mkSgpAQ4dsxm5zXXAOed58jOPXvUcv9+/33TcA+6YZ51D7lE4tprgZo1AQCZ28/DSvin+98v/ARd8B1KGzUJ6xTvvw9kZam5F6JJV3yHjlgmM714KSkJPNdJMrJuHfD1125bETrlQ9zDoDLUE3oeCtAF353xH4XHsajmH3AQ5+thK5bBo0YuQFnV6sANN/hHmJ0NbN0KjwfYtQu4664zehYRmqYwW+9fgG5AlSqRnygAZWXAzp3RjdN8PUUVqmExukSsodHW4F9RCz+hY/CA5YS6dUPrG5UMtGoF/P73blsROqkv7sxqurUQSUMJAOAozsNidDnj/0eMwoQJwBD864xf6YUNzoiR1kmxMXaiA37UI0xPB5o3xz//qTpCzpoVskmWaGKlfVzc4NVXgaZNVQfEaGEWd7uPl5BYHDsGFBa6bYUAAJXcNiDmlJaGdVhF+Krl1ViFkzj7zPan6IcSpAEAOve+G2XeqSw1cd+NxtiNxn7xamOLRYtgOfdwOHIEmD0b6N/fWfilS9Vyzx4l8tHA7nqk9CO18HiUq5T6ShR3Ujvn/v77gUcENDG+yTCfIhgjmbgaG3DlmW201KADAAAcx0lEQVRGBUzCg5iEB+GpXOVMzjmc4UUA4A7MRCeoMTWIgP/7P2fHaeeLZs79vvuAAQOcDc8BAJUrq+WpIEPzhEK0c+6xyPnL30Tk9OkDpKW5bUVqktri/qc/hVS7k1vt8jNFMPmo5fi4sjL4FMuMGROSlQCA2bgDP6LTGZH+xz+cHReLnKxWOes06c46Sy3DFfddu3wntgdil3OPZnqJuEeOcVwuIbqk5s9QXp7zEeR++gno0AEAQFBv692Yiiy0dXy6p57S1ytUAAYOtA8bTFxCzYHHolgmVDRxD7eVRBNvA5gXXtD9zOmQiEKaiDYJ/mh/oMbBQcsDqZlzX7QIOHEieLiOHYHrrjuzqYn7l7gbOxFe4fHMmYH3BytusBL3jRtVuOxs/5xOIlSoauWlkTaBe/NNfT0ZytxF3JODSy9NvBGq40FqinuwNobvvaeWprezAkX+ts6ZE1p4syiXlfluL10KXHEFMGoU0KYN0Lu3734nOXdmNclMQYEzm7S4nIpXLARXO/e0aSr+X3+NTnzRxBinCL2QaKSmuGs1fHZ07qyWBw74eGs593hiFnOj2JeUAFu3qvXMTOvjtQpVZtW8kshfxD/5RBV5/PGPodlWUhJaeDMej/28C8HQxPJf3hanWpl8uB8SLb5YlbknmrgvXAgsWOC2FdHhyBFgwwa3rUg+UlPcg6H9oxUV+XjHQtw//dR32ywu5py7cbtyZf0nw8jChf7xMQPvvKPWzRPcaKJ+/LgzmzVCFXfztY0Yob6jof7NAPZiGamIxkrc3SwWs+Lmm4Fu3dy2IjpcdZWaKEsIjdQT91OngFtuCRymWjWlmsbZsYHw2zEGIFhb8WDFMhs3qqVRlG6+2X9mM2a9SVlJiXWuLdTLC7UM3Sy8W7xt/3fvDi0eY1yJ3JkpWjl36fQTGNMPtuCQ1BP3n38O/KZNmaKWzz6r+hUDap7HGjXwU727AACTJsXYRgOBcu5GzDlOzfTJk9Vy2DD9Q1BUZJ1rc5pr1cJFWiyjxfPkk8C+faEda3cL45FDnjlTTf8ZCuGK+6pVQPXqsR+7hAh4+OHYnkNILByJOxF1J6ItRJRDREMs9vcnosNElO11j0bfVIcE6+p2773+fr16AUePoqyCyvpWrx4Du2wwi9Xf/x7a8VqZPKC3T//zn63DOs25OxF3ZuDdd4H8fGD7duX3wAPAypXW4T/80Nm5jfEbl2b/UAnluJ491TzGocRpvI8bNqi/npMng7f919IrHuXj48cnXvGREDuCvu5EVBHASAC3AmgJ4D4iamkR9HNmbu11/4mync6xE/f77w+qMKG2GXeasy0tVR11iPwHjDS/bB99FNg2J9iN8eK0zN2JuC9frj4ijz3mO3TPjBn6eiSlXLEqlollharHA3zxhSofHjwYqFpVDcfwww9KWLURPAEl/Nu2xb9p5/PPx/d88WLtWuDgQbetSCycvH7XAMhh5h3MfBrAFAA9Y2tWGJw4oZRm8GDr/d26AU8/HTCKYOJu7pxUsaIz09LS9I46Zs47z9l4LOH0ejXz/ffOyr+diHtxsVqamyj+Sx9PLSLhinWZ+86dkRc7mXPu772n/xguW6aW+/erSuWHHwYuvlgPf+edql4/1GankfLuu/E5TyC2bFGDzYV7zadPA7/9pm/feacal89pv8XyghNxbwBgr2E71+tn5i4iWktE04ioUVSsC4UPPwT+8x99FCszDp6kYOL+ySfW4SMl3OFy//3v0I+ZP9+vkZAtVuL38MO6sAPAkiXO4iorUx2wtKEaFiwIXERgJ3rGYzZtCr0sH1A5vKZN7YuvnGLOuTtprjdrlhquX2vxFOa4dlGhpATIzY3tOU6fBp55xnd4iZtuAl55RRXphcN11/kOGaWNsGo7P0I5JVoVqrMANGbmVgAWAPjUKhARDSSiTCLKPHz4cJRO7SVYYaKDwkatrL1SJT3nlcj86U+hHzNwIHDOOaro5tAh6zBacYpVa5nx44P3wtUwfvymT1cdsN58U4042a2bGjTqqqusj3XSFLJlS6BhQ3375El1m7UiErvjtEfP2KQ0HAK1ltm0yfqYPn18i7E0cY9nKyCtHuCPfwQaNXLWmTtcvvxSNYk1fki185tbhjklK0stmf3HYDL2q9i1S68PKo84Efd9AIw58YZevzMwcz4za1VH/wHQzioiZh7NzBnMnFG3bt1w7LUnWDbawdszYoTKUdx2mxqZoKVVzUKKcMklwAUXWNcvByuWcTqKn/GPRPuN3rhRzy1OnQqsX299bKjFMidPqjLuF18E7rjDv8hMO27lSuDqq9V6pC1fs7P1de2jomEnXCdP+m6HUzS0cqUSrnDRJp7QPtLmP7mDB9XzEWgCr+XL9aauTjBWLFuNZLpqlfO4NH791X/0VK1/IqCKQps1Cz3eVMHJ470aQHMiakJElQH0AeCTdyOieobNOwHY5FtiSDBx/8MfgkZx3nmqtYr28P3yi6qENJaJp5rgf/GF+mU2tiXWkvLtt62PSUsL/K08cEAVYRknJDH2EHVSnBVqJyatrfi4cYE7Tf3vf3qxUiTFaj/+6Csk2iyzwTCHCTXnvnatmpXQrg7HCd9+67tNpCp3n35aCe60aerP7oknVM7big4dgMsvD34urX2DsfhJ+/Aaxb19+9D/XhK55U9ZmbtFboCDUSGZuZSIngIwD0BFAOOYeQMRDYOaqHUmgGeI6E4ApQAKAPSPoc2h07Ch4zlJjVSqpIpqVq3Sy3YTqVwvWnNVaj9R5i76dmXIlSsrkbTissuAevX8/Y1DIq9ZE9wmTYCtytwzMuybKlq98MzA2LH+/k7Fff9+NfKlcfq4nBz/84ZTzBCquL/xRuD9a9c6a8Zpplcvdb8ff9w3DX/+WU0JGS6auBvTxm4OgrKywC2ZS0t99yeyuGdkqD87NzvdORryl5nnAphr8htqWH8RwIvRNS1EAr2pEdZ81q6tv9hvv60msygvfPut3u9Lo3JllUO2wu5XXXvIly1zVg7aqJEqMrIqllmzxv8DoYWzGhztyy99K4E1nBbLNPA2Hzh9Gti8WdUTmIX8P/8B/vtfZ/EZCVQsM3q0KtoyCnqg6XCXLQOuv975uY1pq9lRqZKqANWINAOh5dKtmsia07CsTIWfPFl9bM4+29fGceN8W6xF2toplhiL7NwidXqoBhLwKA4roPUMbdEialEmDDfcADzyiF5hBQC33uo/Pk44U6JpL2koFVyff+7/92CVE5owIXA8dhXHwb75hw/7VgSOHKnuf4UK/r/c4Qg74J9z37lT/0A+/rhv01IgsLhbzZxVWhr8T8nj0f9EzHUVVml0++2B4zNi1VxYex3NaVhWphq7PfCASvcRI3xfXfMwDcbK9HiydSuwYoU75w6F1JmsI07ibhy/JdX4/nvlghFO8UO4zd7MWJVjDhgQuAOLXS9Rq8di1Ch9/amnVJ2Ehlb5q/09GAn399v4HGlNNAHrgbLmzLGeuWj6dKBLF+v70r6978faCm0eU8BfjI8cUR3vzj5blfUDwNy5cIxVRsBO3EtL9ZY7VpW5zM6b8TrtsPfJJ0CPHvrfWSCeekql50MP6fYkMqmTcze/bf366evGWSAiJJXF3SluDnRll0s3zoZlxq5oISsLOHrU1884LLJ5nzH/MHq07z67Vj/B0ARu9GjgwgsDx9ejh/9HcsEC1VagSxfrD5+dsJ84obc9N5Zdm/9yxo1TcV93XeCP+rFjwKBBvn7z5lnX2wTKuQfqGPjVV6oZrxOcCO/Bg8pmp3UUI0fqwp4MpIa4HzyoqviN9O6t2jM+9hhw991RO5X2Ar7yiq9/uB2RkpEePdw7t1lUNQLNxRlIlOrXt99nbgcfi6ECnLaosKs81AaIy8oK7Y/KWGfSyNDQuW9f+2MCfcDeeMO3kx8z0L27b7GW1uz27LPV0tzVJZi4//ST/T4zVvfq8ceV//z5avIarS+C0Q5m3/GatGG07QiW0dm1SzV/PXZMFXmGOux2RDCzK65du3YcNa67TmuJprtvvole/AGoWlWdjpn5+uv9zRDnnluyRN2XN94IHG7QIOarr2YuKQkcbuDA6Nv4yCPBwzAzP/ectb9x+/LLY5uea9f6n5OZ+fRp/7CrVtlfy+9/r9YnTPDdd+AA88KFkdvJzHz0qP3+bt18t+vUYV65Uh337rvKb/Vq/2s1uxUr1HLmTOa8PLW+dKn/cbfeyvy3v6n1N96IXHOgWikimEuNnPvy5f5+MRib3YqNG/XhYWvVUsvp04FOnXy7SEcLmbTAOVr5ebAc96hRqteosbw9Xjgp3vvnP63HhDHnAjdvjo5Ndrz8MnDjjb5+jz1mPTaSsReumapV1XLtWl//0tLwe60ayc/XR0i1wjgUgrbdvr16l7Uerk7GYNIqVRcs0P8qOnf2v1fffKPXJTjtABgNkl/c7QbHiFMj2IsvVr+fgN6S5qKL1EMSqKggXGLxwUhVtIpUp8Upw4YF3h+LR8qJuL/8srV/jRrRtSUYM2f6jyX0n/9Yv4KBmgJqxTJmEfz11+Dz7DihTp3AnQ3tKvfXrdOLyUIV4bPO0tetxiwaMUIt7Ya+igXJL+52A2O4UOs3dKjKsbRtq7YDNVsLFy3XE4xwOrKkGtOnq6aXTn/igg135Ja4JyN2o5jWrq2Lu5lFiwLHGa2fcbtceZ8+vu39nTJihPN3febM6PydOCH5xd0upWI5GpINlSqpnmnG7UgwDhGrEUzctWFPL7lEnyq2PDNmTPQqQmPxUoYyPksqUFBgP2eB3WjdGvH4GdeGZogk5x6MZs3sJ7yPJskv7nbNDVwQdzPBxD1QT9cxY3x79WkYWzZYwezvN3Jk4GNSGeboiXssfqnNc+EKiYGTjJlxCI5gHcWMRWi7dgEffBCWWSGR/OJu14i5ffv42mFBsMk8HnvM2r9+feDRR4HWrf33aSMa2qHl7E+dUu2CH31UNQGLFr17Ry+ueBBNcS9PzV3LO05y7saRLJ99NrT4K1cOLXw4JL+4Ww0aAiSFuJtbHmj06aOvm2v2rQbl0njySeUANcTuFVeoPwCnM0Y5YeBA3/5hyUCcGk4JKYTVH3A0icf0isn/2FvNyJwIo/ZAH0v6iy98H5bMTPumYtOn+3aoNf4eDhoUuIfeRx+p3oSA9Tjt0SAtTf/AdOwYm3NEk2jm3IXyQ7TL982jycbjmUzusWVmzQJeeMHX7/hxfUollzn3XOscQKNGwPnn+/s//LA+kYKG8SH4+OPgFTFNmwbOdfTsCXz9deA4ANXK5JJL/P2rV9fHaf/xR1Xpu3dv4g6/ajcmvSAEwuNRc7MmM8mbcz92zDr1E0TYA2FXTNLTYtpxTdy1pTYs/R136CNTHj/uX3xjh9N2xDVrqoGq7r9f93v9daBNG32e8WrVVOXQ6dPAgw86i1dIfPLy3LbAfUpLfSebiTZx+Zt00o01Fi7i4Qdyc+37HicotWsrE48c0f00s0+ftj7mxAm1v0UL3W/aNOaCAuaTJ5mPHXN2bu08ZWX6+oIFbNsF/vhxddypU7pfXp7y27ZNbTdtan0OgHnAgPC6jvfvH3n380R3d9xhv69TJ/ftO37cfRvCceecw1yrlvt2OHH/+Iez99b6XU7l4QfKysKb9t5lli8HXnpJH6bAiF3tfLVqwMSJvoNY3XWXysFXqaKKfkLBWLl4002qCdeYMf4deLS/C2OtvjZjk5ORMV98MfTcSbVqyhbjjEepiLEvhJlEKN5y2lEu0XjoIeuhEBKRl16K/TmSU9xffDEhWsOESvPmarZ2o+h9/70aOyQQffsGHr3QCW3aAP37+/u3bavsMX4khg+3fsE1wXci7s2b+wpVfr6afEPDqpnniBGqAtlqSjwru6NNvIThnnvsJ3tIBHGvWNF3NqZooY0HHyuqVo3e1JOxxG6KyqjjJHsfCxd2sUyg4d6A8OIsZ9gl1cSJzDt3+vuPHs08Z46+rRUVDRjgG+7BB/3j1kZSZGYuLWVu147566+Zf/vN97YtXqwfc+BA8N/aXbuYf/6Z+dJLrfe3aBH4+H79/P3mzHH+W52Wpq8HK4K64grf7awsdZ3LlzOff77vvmuuiex3v379yI433rvZs5n372du0ybyOAHmvn39/UItvrMb4bNRI1WcFOuRMSN1NWqE8KLaAIfFMkEDxMqFLe5Tptin3DnnhBdnOWP5ciWMkbBnjyqPN2IUbA2PRzkrtLCZmYH3a+6ll/R17SP022/Mf/2rf9iTJwO/ZB4P81NPMb/9tu737bfOX9LKlX2v1S7cyZPMhw4xv/IK85//rPx27PC9To9H+V9+OXNGRmTikZWllh9/HNpx69f73zuNoUNDt6NmTX29f381LG5hIfMLL/iG++qr0OJdvdraf9w4ZWvTppGlX6zdli0BXytHpK64B8peGWsqBVewEwgrtm0L/JHR4rr/flWJzMz8738rv8JCPdy8ecpv8GD9GE0wrdyCBdbn0cS9VSvd709/0tdnz2b+3/+ULc2a+V6rXY7RSGkp86ZN1te6aJH6CGi55Hfe0ePo3t25eBjPZR63fPdu5jFj1B+a0V+r3Hv5ZeZbbvG3beVK63MRqY+sMb0AdY3TpunbAwf62vXWW8wffsj83/8qv9279bCffsr8xRfM48fbX9/Wrf7+o0apfQ0bhi+80XQtWwZ/HsIldcV96dLgT7bgGhs26JNkRIM5c/z/EKw4fFgtP/hAtUpitn5EzjvP+hxffaUm63jiCd+GWMzMI0eq9a1b9WPMYY4cUZM31KrFfMMN4T+SCxYogfrtN+YGDfQ4AglJWRlz+/bMM2ZYxwmoj4ZGaWl49hnPaZx04u67lV+HDupviNlX3DU/J3Fb+dWtq5bGFmNasZs2T8+HHyp/czGXE2eehMX8sdLcCy/ofziPPab7Z2er569FC/WRGTVK/dmKuIeK9t9pdrt3hxefkLJYPSZOS+7mz9dnHvJ4VNNTM8uX23/IjDN0hcuePczTp6t1zf6HH1bCee21zgWjuFh9uIyEIzZt26pjGjTw9R8+XPmvWqX7aTMTAao4KhjvvaeagVrZuG+fSmvtA87M3Lq12qf94Wj1GE2aqO38fPV398sv6g/xvvuUv1Ux0KuvMj/9tFr3eNTfjdWzo/1pZGWpDEdxsXpO7PjtNz3d09KY77wzeDo4IXXFXWtkbXRDhoQXl5DSWL2gRPE5d2Gh3lcgGmRnq1yjkUhyg717q4rtUDh0SJ3vo498/cvKmNet8w//2msq/N/+Fp6NgL0gpqf7irrGtm1K8M2UljIXFal17U9j0iS1NPcxsRP3Tz8N7zqiTVTFHUB3AFsA5AAYYrH/LACfe/evBNA4WJxhi7sxS6C5qVPDi0tIaTp3Zh42TLXimT07MjFMRHbs8K+cjTV2ne2sOHJE1RccPBh9O7Q5SffvD/3Y4mLV2irQ/vff958TeezY8O2NJk7FnVRYe4ioIoCtAG4GkAtgNYD7mHmjIcwTAFox8yAi6gOgFzMHHLoqIyODM8MZsb6wUA0x0L69mn+MGejRQ0aHEoJCpLpIvP6625YIkVJWpjreXXhh7M7RqZMaP0lj377I+5tEAyJaw8wBusIpnAwcdg2AHGbe4Y14CoCeADYawvQE8Ip3fRqAj4jUD3BIVjuhWjU1I23bttZdPQXBhhg8jYJLVKwYW2EHgE8/BSZNArp1U/MjJ1vPXSfi3gDAXsN2LgBz99AzYZi5lIiOAagNwOFwViFy000xiVYQBEHjkkvUvMjJSlyHHyCigUSUSUSZh4PNRiwIgiCEjRNx3wfAOHNnQ6+fZRgiqgSgBoB8c0TMPJqZM5g5o642CpUgCIIQdZyI+2oAzYmoCRFVBtAHwExTmJkA+nnX7wawKCbl7YIgCIIjgpa5e8vQnwIwD0BFAOOYeQMRDYNqkjMTwFgAnxFRDoACqA+AIAiC4BKOptlj5rkA5pr8hhrWiwHcE13TBEEQhHBJzvHcBUEQhICIuAuCIKQgIu6CIAgpSNDhB2J2YqLDAHaHeXgdxKqDVGQkql1A4tomdoWG2BUaqWjXxcwctC25a+IeCUSU6WRshXiTqHYBiWub2BUaYldolGe7pFhGEAQhBRFxFwRBSEGSVdxHu22ADYlqF5C4toldoSF2hUa5tSspy9wFQRCEwCRrzl0QBEEIQNKJOxF1J6ItRJRDREPifO5GRLSYiDYS0QYiGuz1f4WI9hFRttfdZjjmRa+tW4jolhjatouI1nnPn+n1q0VEC4hom3d5ntefiOhDr11riahtjGy6zJAm2UR0nIiedSO9iGgcER0iovUGv5DTh4j6ecNvI6J+VueKgl1vEdFm77mnE1FNr39jIjppSLdRhmPaee9/jtf2iKYms7Er5PsW7ffVxq7PDTbtIqJsr38808tOG9x7xpzMxZcoDmrgsu0AmgKoDOAXAC3jeP56ANp616tDTT/YEmoWqr9YhG/ptfEsAE28tleMkW27ANQx+b0J75y3AIYAGO5dvw3ANwAIwLUAVsbp3h0AcLEb6QWgM4C2ANaHmz4AagHY4V2e510/LwZ2dQNQybs+3GBXY2M4UzyrvLaS1/ZbY2BXSPctFu+rlV2m/e8AGOpCetlpg2vPWLLl3M9M+cfMpwFoU/7FBWbOY+Ys7/oJAJugZqGyoyeAKcx8ipl3Qk0gfk3sLfU5/6fe9U8B/N7gP5EVKwDUJKJ6MbalK4DtzByo41rM0ouZl0KNWGo+XyjpcwuABcxcwMy/AlgANXl8VO1i5vnMXOrdXAE1h4ItXtvOZeYVrBRiouFaomZXAOzuW9Tf10B2eXPfvQFMDhRHjNLLThtce8aSTdytpvwLJK4xg4gaA2gDYKXX6ynv79U47dcL8bWXAcwnojVENNDrdwEz53nXDwC4wAW7NPrA96VzO72A0NPHjXR7GCqHp9GEiH4mou+J6HqvXwOvLfGwK5T7Fu/0uh7AQWbeZvCLe3qZtMG1ZyzZxD0hIKJqAL4E8CwzHwfwMYBLALQGkAf1axhvOjFzWwC3AniSiDobd3pzKK40jSI1ycudAKZ6vRIhvXxwM33sIKKXAJQCmOT1ygNwETO3AfAcgP8R0blxNCnh7puJ++CbgYh7ellowxni/Ywlm7g7mfIvphBRGtTNm8TMXwEAMx9k5jJm9gAYA70oIW72MvM+7/IQgOleGw5qxS3e5aF42+XlVgBZzHzQa6Pr6eUl1PSJm31E1B9ADwAPeEUB3mKPfO/6Gqjy7Eu9NhiLbmJiVxj3LZ7pVQnAHwB8brA3rullpQ1w8RlLNnF3MuVfzPCW6Y0FsImZ3zX4G8urewHQavJnAuhDRGcRURMAzaEqcqJt1zlEVF1bh6qQWw/f6Q/7AfjaYNdD3hr7awEcM/w6xgKfHJXb6WUg1PSZB6AbEZ3nLZLo5vWLKkTUHcALAO5k5iKDf10iquhdbwqVPju8th0nomu9z+hDhmuJpl2h3rd4vq83AdjMzGeKW+KZXnbaADefsUhqiN1wULXMW6G+wi/F+dydoH6r1gLI9rrbAHwGYJ3XfyaAeoZjXvLaugUR1sgHsKspVEuEXwBs0NIFQG0A3wHYBmAhgFpefwIw0mvXOgAZMUyzc6AmS69h8It7ekF9XPIAlECVYz4STvpAlYHneN2AGNmVA1Xuqj1jo7xh7/Le32wAWQDuMMSTASW22wF8BG8HxSjbFfJ9i/b7amWX138CgEGmsPFMLzttcO0Zkx6qgiAIKUiyFcsIgiAIDhBxFwRBSEFE3AVBEFIQEXdBEIQURMRdEAQhBRFxFwRBSEFE3AVBEFIQEXdBEIQU5P8BHpDceWK0hhsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe1d4350a50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##################################################################################################\n",
    "def AddLeNetModel(model, data):\n",
    "    # Image size: 28 x 28 -> 24 x 24\n",
    "    conv1 = brew.conv(model, data, 'conv1', dim_in=1, dim_out=20, kernel=5, \\\n",
    "                weight_init=('MSRAFill', {}),\n",
    "                bias_init=('GaussianFill', {}))\n",
    "    # Image size: 24 x 24 -> 12 x 12\n",
    "    pool1 = model.net.MaxPool(conv1, 'pool1', kernel=2, stride=2)\n",
    "    # Image size: 12 x 12 -> 8 x 8\n",
    "    conv2 = brew.conv(model, pool1, 'conv2', dim_in=20, dim_out=50, kernel=5, \\\n",
    "                weight_init=('MSRAFill', {}),\n",
    "                bias_init=('GaussianFill', {}))\n",
    "    # Image size: 8 x 8 -> 4 x 4\n",
    "    pool2 = model.net.MaxPool(conv2, 'pool2', kernel=2, stride=2)\n",
    "    # 50 * 4 * 4 stands for dim_out from previous layer multiplied by the image size\n",
    "    fc3 = brew.fc(model, pool2, 'fc3', dim_in=50 * 4 * 4, dim_out=500, \\\n",
    "                weight_init=('MSRAFill', {}),\n",
    "                bias_init=('GaussianFill', {}))\n",
    "    fc3 = model.net.Relu(fc3, 'relu3')\n",
    "    pred = brew.fc(model, fc3, 'pred', 500, 10)\n",
    "    softmax = model.net.Softmax(pred, 'softmax')\n",
    "    return softmax\n",
    "###################################################################################################\n",
    "\n",
    "\n",
    "#TRAINING and TESTING\n",
    "train_model = model_helper.ModelHelper(name=\"mnist_train\", arg_scope=arg_scope)\n",
    "data, label = AddInput(\n",
    "    train_model, batch_size=64,\n",
    "    db=os.path.join(data_folder, 'mnist-train-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "\n",
    "softmax = AddModel(train_model, data)\n",
    "AddTrainingOperators(train_model, softmax, label)\n",
    "\n",
    "workspace.ResetWorkspace()\n",
    "workspace.RunNetOnce(train_model.param_init_net)\n",
    "workspace.CreateNet(train_model.net, overwrite=True)\n",
    "total_iters_m = 2000\n",
    "accuracy_m = np.zeros(total_iters_m)\n",
    "loss_m = np.zeros(total_iters_m)\n",
    "printmd('**Training status: Running**  Wait till the process is completed')\n",
    "for i in range(total_iters_m):\n",
    "    workspace.RunNet(train_model.net)\n",
    "    if(i%100 == 0):\n",
    "        print \"    Completed iterations:\", i, \", Total iterations to be completed:\", total_iters_m, \\\n",
    "        \", Current Loss = \", workspace.blobs['loss']\n",
    "    accuracy_m[i] = workspace.blobs['accuracy']\n",
    "    loss_m[i] = workspace.blobs['loss']\n",
    "#print \"Training completed\"\n",
    "printmd('**Training status: Completed**')\n",
    "\n",
    "#print \"Testing status: Running\"\n",
    "printmd('**Testing status: Running**  Wait till the process is completed')\n",
    "test_model = model_helper.ModelHelper(\n",
    "    name=\"mnist_test\", arg_scope=arg_scope, init_params=False)\n",
    "data, label = AddInput(\n",
    "    test_model, batch_size=100,\n",
    "    db=os.path.join(data_folder, 'mnist-test-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "softmax = AddModel(test_model, data)\n",
    "AddAccuracy(test_model, softmax, label)\n",
    "workspace.RunNetOnce(test_model.param_init_net)\n",
    "workspace.CreateNet(test_model.net, overwrite=True)\n",
    "test_iters_m = 1000\n",
    "test_accuracy_m = np.zeros(test_iters_m)\n",
    "for i in range(test_iters_m):\n",
    "    if(i%100 == 0):\n",
    "        print \"    Completed test set:\", i, \", Total sets to be tested:\", test_iters_m\n",
    "    workspace.RunNet(test_model.net.Proto().name)\n",
    "    test_accuracy_m[i] = workspace.FetchBlob('accuracy')\n",
    "#print \"Testing completed\"\n",
    "printmd('**Testing status: Completed**')\n",
    "# After the execution is done, let's plot the values.\n",
    "pyplot.plot(loss_m, 'b')\n",
    "pyplot.plot(accuracy_m, 'r')\n",
    "pyplot.legend(('Loss', 'Accuracy'), loc='upper right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Effects of Initializers**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" cellpadding=\"3\" cellspacing=\"0\"  style=\"border:black; border-collapse:collapse;\"><tr><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Initializer&nbsp;Type</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Num&nbsp;iterations</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;LR</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;Loss</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Loss</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Init&nbsp;Training-Acc(%)</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Training-Acc(%)</b></td><td  style=\"background-color:LightGray;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\"><b>Final&nbsp;Testing-Acc(%)</b></td></tr><tr><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">Constant&nbsp;-&nbsp;high</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">43.1735</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">41.7344</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">6.2500</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">9.3750</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.3200</td></tr><tr><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">Constant&nbsp;-&nbsp;zero</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2.3026</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2.3023</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.9375</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">10.9375</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">11.3500</td></tr><tr><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">Xavier</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">3.2040</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.1746</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">12.5000</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">92.1875</td><td  style=\"background-color:Ivory;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">94.2800</td></tr><tr><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">MSRA-Gaussian</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">2000</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.0010</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">3.0503</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">0.1932</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">12.5000</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">92.1875</td><td  style=\"background-color:AliceBlue;border-left: 1px solid;border-right: 1px solid;border-top: 1px solid;border-bottom: 1px solid;\">94.0900</td></tr></table>"
      ],
      "text/plain": [
       "<ipy_table.ipy_table.IpyTable at 0x7fe1d42b5e10>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "printmd('**Effects of Initializers**')\n",
    "\n",
    "initializers = [\n",
    "    ['Initializer Type', 'Num iterations', 'Init LR', 'Init Loss', 'Final Loss', 'Init Training-Acc(%)', \\\n",
    "     'Final Training-Acc(%)', 'Final Testing-Acc(%)'],\n",
    "    ['Constant - high', total_iters_c_h, base_lr, loss_c_h[0], loss_c_h[total_iters_c_h-1], accuracy_c_h[0]*100, \\\n",
    "     accuracy_c_h[total_iters_c_h-1]*100, np.mean(test_accuracy_c_h)*100],\n",
    "    ['Constant - zero', total_iters_c_m, base_lr, loss_c_m[0], loss_c_m[total_iters_c_m-1], accuracy_c_m[0]*100, \\\n",
    "     accuracy_c_m[total_iters_c_m-1]*100, np.mean(test_accuracy_c_m)*100],\n",
    "    ['Xavier', total_iters_x, base_lr, loss_x[0], loss_x[total_iters_x-1], accuracy_x[0]*100, \\\n",
    "     accuracy_x[total_iters_x-1]*100, np.mean(test_accuracy_x)*100],\n",
    "    ['MSRA-Gaussian', total_iters_m, base_lr, loss_m[0], loss_m[total_iters_m-1], accuracy_m[0]*100, \\\n",
    "     accuracy_m[total_iters_m-1]*100, np.mean(test_accuracy_m)*100],\n",
    "\n",
    "];\n",
    "make_table(initializers)\n",
    "apply_theme('basic')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
